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Inferring  Examinee  Ability  When  Some  Item  Responses  Are  Missing 


Abstract 

The  basic  equations  of  item  response  theory  (IRT)  provide  a 
foundation  for  inferring  examinees'  abilities  and  items' 
operating  characteristics  from  observed  responses.  In  practice, 
though,  examinees  will  usually  not  have  provided  a  response  to 
every  available  item- -for  reasons  that  may  or  may  not  have  been 
intended  by  the  test  administrator,  and  that  may  or  may  not  be 
related  to  examinee  ability.  The  mechanisms  that  produce 
missingness  must  be  taken  into  account  if  correct  inferences  are 
to  be  drawn.  Using  concepts  introduced  by  Rubin  (1976),  ,wc-Jhl‘> 

discussSthe  implications  for  ability  and  item  parameter 

ft 

estimation  that  are  entailed  by  alternate  test  forms,  targeted 
testing,  adaptive  testing,  time  limits,  and  omitted  responses.' 


r\ 
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Introduction 


The  capability  to  measure  different  examinees  with  different 
test  items  is  an  oft-cited  advantage  of  item  response  theory 
(IRT) .  This  option  implies  a  problem  of  inference  in  the  presence 
of  missing  data,  since  an  examinee  may  not  have  provided  a 
response  to  every  item  in  the  complete  item  set.  Five  types  of 
missingness  are  in  fact  encountered  regularly  in  routine 
applications  of  IRT’ 

Case  1:  Alternate  test  forms.  Two  or  more  tests  with  similar 
content  but  different  items  are  often  employed  to  minimize  carry¬ 
over  effects  (as  in  test-retest  designs),  reduce  fatigue  and 
practice  effects  (by  splitting  a  test  into  shorter  subtests),  or 
avoid  cheating  behavior.  A  examinee  is  typically  administered 
one  form  selected  at  random. 

Case  2:  Targeted  testing.  Two  or  more  tests  with  similar 
content,  but  pitched  at  different  levels  of  difficulty,  can  be 
used  to  make  testing  more  efficient  when  background  information 
(such  as  grade  or  courses  taken)  is  available  for  deciding  which 
test  to  administer  to  each  examinee. 

Case  3:  Adaptive  testing.  Testing  can  also  be  made  more 
efficient  and  less  time-consuming  if  each  item  presented  to  an 
examinee  is  selected  on  the  basis  of  his  responses  up  to  that 
point,  and  possibly  background  information  as  well. 
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Case  4:  Not-reached  items.  Under  typical  testing  conditions 
some  examinees  will  not  reach  the  last  few  items  on  a  test 


because  of  the  time  limit. 

Case  5:  Omitted  items.  Even  when  an  item  has  been  presented  to 
an  examinee  and  he  has  time  to  reach  it,  he  will  sometimes  choose 
not  to  respond. 

When  incomplete  data  of  any  of  these  types  are  encountered, 
the  IRT  model  that  presumably  accounts  for  the  responses  that  are 
observed,  is  embedded  in  a  more  encompassing  model  that  determines 
which  responses  will  be  observed  and  which  will  be  missing.  This 
paper  discusses  the  implications  that  missing  responses  hold  for 
likelihood  and  Bayesian  inferences  about  examinee  ability 
parameters  and  item  parameters,  assuming  an  IRT  model  holds.  When 
can  the  process  that  causes  missingness  be  ignored?  When  it 
cannot  be  ignored,  how  can  it  be  modeled?  How  can  conventional 
IRT  methods  for  missing  responses  be  evaluated  in  this  framework? 

The  following  section  extends  IRT  notation  to  handle 
missingness,  using  concepts  and  notation  from  Little  and  Rubin 
(1987)  and  Rubin  (1976).  Next,  Rubin's  (1976)  conditions  for 
when  the  missingness  process  can  be  ignored  are  reviewed.  Each 
of  the  five  types  of  missingness  listed  above  are  then  discussed 
in  some  detail  in  the  problem  of  inferring  ability  when  item 
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parameters  are  known.  This  is  followed  by  the  extension  to  item 
parameter  estimation.  A  final  section  summarizes  our  results. 


Background  and  Notation 

At  the  heart  of  IRT  is  the  model  for  the  response  to  item  j , 
with  its  possibly  vector-valued  parameter  /3j ,  from  an  examinee 
with  ability  9.  The  Rasch  model  for  dichotomous  items,  for 
example,  posits 


P(Uj  =  -  explu^S  -  b  )]/[!  +  exp(0  -  b^ ) 


where  u^-1  denotes  a  correct  response  and  u^-0  an  incorrect  one, 
and  bj  is  the  difficulty  parameter  of  item  j .  We  assume  IRT 
functions  that  are  twice  differentiable,  and  interpret 
P(IL  -  l|0,/)j)  as  the  proportion  of  correct  responses  we  would 
expect  to  many  items  with  /?  -  0  from  many  examinees  with  that 
value  of  9 . 

Under  the  usual  assumption  of  local  independence,  the 
conditional  probability  of  the  response  vector  U  - 
for  n  items  is  obtained  by  the  product  rule: 


P(U  =  u|0,0) 


n  P(u.  -  u. 
j-1  J  J 


(1) 
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It  is  further  assumed  that  if  y  denotes  background  information 
about  an  examinee  such  as  age,  GPA,  or  courses  taken,  then 


P(U  =  u|0,/J,y)  =  P  (U  -  u|  6,p)  . 

When  there  is  no  possibility  of  missing  responses,  (1)  can 
be  interpreted  as  a  likelihood  function,  say  L(fl|u),  once  a 
particular  value  u  of  U  has  been  observed.  Direct  likelihood 
inferences  are  based  solely  on  relative  values  of  L  at  different 
values  of  8.  It  might  be  said,  for  example,  that  the  probability 
of  u  is  twice  as  high  at  9'  than  at  9''.  The  maximum  likelihood 

a  ~ 

estimate  (MLE) ,  8,  is  the  value  at  which  u  has  the  highest 
probability.  Note  that  in  direct  likelihood  inference,  the  MLE 
concerns  only  the  data  that  were  actually  observed. 

The  role  of  the  MLE  in  sampling  distribution  inferences 
concerns  its  distribution  under  repeated  sampling  of  observations 
with  a  fixed  "true"  parameter  value.  If  n  is  large,  the  sampling 
distribution  of  8  as  computed  from  repeated  observations  of  U  can 
be  approximated  by  a  normal  distribution  with  mean  9  and  variance 


where  i(#|u)  =  log  L(fl|u).  By  considering  the  distribution  of  8 
over  hypothetical  draws  from  the  sample  space,  sampling 
distribution  inferences  involve  datasets  that  could  have  been 
observed,  but  were  not. 
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Bavesian  inferences  are  based  on  the  posterior  distribution 
for  8  given  u,  or 


P(*|u)  -  K  L( 0 | u)  p(0)  ,  (2) 

where  K  is  a  normalizing  constant  and  p(0)  conveys  knowledge 

about  6  before  a  value  of  U  is  observed.  The  posterior  mean  and 

mode  of  8  are  sometimes  taken  as  point  estimates  in  IRT.  The 

2 

posterior  variance  is  approximated  by  a  when  n  is  large.  (This 
is  the  variance  of  the  posterior  distribution  for  8  induced  by  the 
data  actually  observed,  in  contrast  to  the  variance  of  an 
estimator  over  hypothetical  repeated  observations) . 

In  many  applications  of  IRT,  an  examinee  provides  responses 
to  only  a  subset  of  the  n  items  to  which  responses  could  have 
been  observed.  The  data  thus  consist  of  (i)  the  identification 
of  the  subset  of  items  to  which  responses  are  observed  and  (ii) 
the  responses  to  those  items.  The  first  inferential  problem  we 
address  is  to  estimate  an  individual  examinee's  8  from  this 
extended  observation,  assuming  that  both  the  IRT  model  and  the 
item  parameters  are  known.  To  this  end,  we  adapt  notation  from 
Little  and  Rubin  (1987)  and  Rubin  (1976)  in  defining  the 
following  terms: 


o 


I 


U  -  (U^,...,U  )  is  the  (hypothetical)  random  vector  of 
responses  to  all  items  in  the  full  item  set. 
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o 


M  -  (M^,...,M  )  is  an  associated  "missing-data  indicator," 

with  each  element  taking  values  of  0  or  1 .  If  m  -  1,  the 

value  of  U.  will  be  observed;  if  m.  —  0,  the  value  of  U.  will 

J  J  J 

be  missing. 


o  V  =  (V^,...,V  )  conveys  the  data  that  are  actually  observed: 

V.  =  U.  if  m.  =  1  but  V,  »  *  if  m.  =  0. 

J  J  J  J  J 


An  observed  value  of  M,  say  m,  effects  a  partition  of  U,  u, 

V,  and  v  according  to  which  elements  are  observed  and  which  are 

missing.  That  is,  we  may  write  U  -  (U  .  ,U  .  )  to  distinguish  the 

mis  obs 

missing  and  observed  elements  of  U,  respectively.  Similarly,  u  = 


(u  .  ,u  ,  ),  V  -  (V  .  ,V  ),  and  v  -  (v  .  ,v  ,  ) 

mis  obs  mis  obs  mis  obs 


m,  let  v  denote  a  realized  value  of  V. 


As  with  u  and 


Example 

An  examinee  is  administered  a  two- item  test.  With  each  item 

2 

scored  right  or  wrong  (1  or  0),  there  are  2  -4  possible 

patterns  for  U:  (0,0),  (0,1),  (1,0),  and  (1,1).  The  second 
response  may  be  missing,  however.  With  1  representing  "observed" 
and  0  representing  "missing,"  there  are  4  conceivable  patterns 
for  M,  of  which  (1,0)  and  (1,1)  can  be  realized.  If  the  examinee 
would  have  responded  incorrectly  to  the  first  item  and  correctly 
to  the  second,  but  the  response  for  the  second  item  is  missing, 
then  u  =  (0,1),  m  -  (1,0),  and  v  -  (0,*).  # 
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Inferences  must  of  course  be  based  on  the  data  that  are 


actually  observed,  namely  realizations  of  V  *  (U  ^  .M).  Modeling 
the  hypothetical  complete  data  vector  (U,M)--even  if  there  is  no 
intention  of  observing  a  response  to  every  item- -is  a  convenient 
way  to  begin.  It  forces  us  to  explicate  our  beliefs  about  the 
relationships  among  ability,  item  response,  and  missingness- - 
exactly  what  is  required  for  building  a  sensible  model  for  V. 
Recalling  that  p(u,m)  can  be  written  as  p(m|u)  p(u)  or  as  p(u|m) 
p(m) ,  define  the  following  densities: 


o  f^(u)  t*le  density  for  all  n  responses.  In  this  paper, 

fff(u)  takes  the  form  shown  in  (1),  so  by  local  independence 
we  can  write  f  (u)  -  f  (u'>  f  (u'')  for  any  ordering  and 

V  u  u 

partitioning  of  the  items  into  (u' ,u' ')-- including 

(u  .  ,u  )  . 
mis  obs 

o  g^(m|u)  is  the  probability  that  M  takes  the  value  m  = 

(m^,...,m^)  given  that  U  takes  the  value  u  -  (.u^ ,  .  .  .  ,  u^) , 
with  4>  being  the  (possibly  vector-valued)  parameter  of  the 
missingness  process.  It  is  possible  for  8  to  be  a  component 
of  <j> ,  in  which  case  the  value  of  8  itself  plays  a  role  in 
determining  whether  a  response  will  be  observed.  In  these 
cases  we  shall  sometimes  write  g(m|u,0,<^)  to  emphasize  the 
dependence  on  8 . 


o 


i 


h^(u|m)  is  the  probability  that  U  takes  the  value  u  given 
that  M  takes  the  value  m. 

o  t^(m)  probability  that  M  takes  the  value  m.  Again,  6 

may  be  a  component  of  <f> . 


Example  (continued) 

Suppose  that  the  missingness  process  in  the  two- item  example 
initiated  above  can  be  described  as  follows:  The  second  response 
is  observed  whenever  the  first  response  is  correct;  the  second 
response  will  be  observed  with  probability  <f>  if  the  first  response 
is  incorrect.  Then 


g^(m|u) 


1  if  m=(l,0)  and  u=(l,0)  or  (1,1) 

l-<f>  if  m=(l,0)  and  u-(0,0)  or  (0,1) 

<t>  if  m=(l,l)  and  u=(0,0)  or  (0,1) 

0  otherwise .  # 


Whenever  not  all  potential  responses  may  be  observed  for  any 
reason- -even  if  they  all  do  turn  out  to  be  observed- -  the  data  are 
v.  To  obtain  the  likelihood  function,  we  start  with  the 
likelihood  for  the  (hypothetical)  complete  data  (U,M),  then 
average  over  the  missing  responses  u  .  : 


8 


L(«,*|v)  -  5 [ (<? ,<^) ,n  ]  Jf  (u 


,u  ,  )g,(m|u  . 

mis  obs  °4>  '  mis 


u  ,  )du  . 
obs  mis 


where  6  takes  the  value  1  if  a  value  (8  ,<f>)  is  in  the  parameter 
space  0  and  0  if  it  isn't.  This  observed-data  likelihood  is  a 

u<p 

weighted  average  over  all  complete-data  likelihoods  that  have  the 

targeted  responses  to  the  observed  items.  The  weights  are 

proportional  to  the  probabilities  of  these  potential  response 

patterns  for  the  different  values  u  .  ,  given  m  and  u  ,  Using 

mis  obs 

local  independence,  we  can  bring  the  probability  for  the  observed 
responses  outside  the  integral: 


u».^|v)  -  SC.-)  V%bs>/VVs)V”|u*is-'Wdu,»is  ■  ‘3> 


Equivalently,  using  the  alternative  expression  for  p(u,m), 


L(*’*|v>  -  «V>  ym)  Jyumls,uobs|m)  dum.s  .  (4) 


Appropriate  likelihood  inferences  are  based  on  relative  values  of 
L(0,^>|v)  at  various  values  of  (8  ,<j>)  ,  or  at  various  values  of  8 
after  eliminating  <j>  by  conditioning  or  maximizing.  Appropriate 
Bayesian  inferences  are  based  on  the  posterior  distribution 


p(6,<f>\v)  a  L(0,^|v)  p {&,<!>)  , 


(5) 


where  p {8  ,<t>)  conveys  prior  knowledge  about  8  and  <p .  Appropriate 
sampling  distribution  maximum  likelihood  inferences  concern  the 
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distribution  of  (8  ,j>)  from  (3)  or  (4),  over  hypothetical  repeated 

observations  of  V  for  fixed  (8 ,<fr) . 

In  general,  then,  the  correct  likelihood  function  involves  a 

nuisance  parameter  tf> ,  and  depends  not  on  just  the  responses  that 

were  observed,  through  ^^u0^s)*  but  on  the  responses  that  were 

not  observed,  through  f.(u  .  )  and  g,(m|u  .  ,u  .  ). 

-  &  0  mis  i '  mis  obs 

Example  (continued) 

With  IRT  for  binary  variables,  the  integral  over  u  ^  that 
appears  in  (3)  is  a  summation  over  all  possible  response  patterns 
with  uQks  =  u0bs '  our  two'^tem  example  with  the  first  response 

incorrect  and  the  second  response  missing,  the  potential  complete 
patterns  u  are  (0,1)  and  (0,0).  Thus, 

L(*,*|V-(0,*))  =  61(6, 4>), 0^]  ff,(V0) 

x  (ftf(U2-0)  g  [M-(l,0)|U-(0,0)] 

+  ffl(U2-l)  g^[M-(l,0)|u-(0,l)])  .  (6)  # 


Conditions  for  Ignorability 

Ignoring  the  missingness  process  when  drawing  inferences 
about  8  means  that  rather  than  using  the  correct  likelihood 
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L(0,^|v),  using  a  facsimile  of  (1)  as  applied  to  u  alone: 


L  <*|Uobs)  -  «<».«,> 


Hi. a.)  n  P(u  -  u  \t,p  ) 
obs  J  J  J 


(7) 


In  particular,  direct  likelihood  inferences  about  8  that  ignore 

★ 

the  missingness  process  simply  compare  values  of  L  at  various 
values  of  8.  Bayesian  inferences  that  ignore  the  missingness 
process  start  with  an  analogue  of  (2) ,  a  psuedo-posterior 
distribution  proportional  to 

L*(9lUobs)  P<J)  ■  (8) 

Sampling-distribution  maximum  likelihood  inferences  that 

A 

ignore  the  missingness  process  consider  the  distribution  of  8  from 
(7)  over  repeated  samples  of  responses  to  the  items  for  which 
m.-l.  This  involves  a  different  reference  sample  space- -not  the 
sample  space  of  v  values,  driven  by  (8  ,4>) ,  but  a  sample  space  of 
Uobs  va^-ues  f°r  a  m>  driven  by  6.  (This  reasoning  is  used 

in  survey  sampling  when  the  exact  size  of  the  sample  is  not  known 
before  it  is  obtained.  Even  though  the  sample  size  N  is  a  random 
variable  with  its  own  distribution  and  parameters,  standard  errors 

A 

for  8  are  typically  computed  with  respect  to  repeated  draws  with 

the  observed  sample  size  N,  rather  than  with  respect  to  repeated 

draws  of  (U  ,  , N) . ) 

ODS 


11 


It  is  a  pleasant  state  of  affairs  when  ignoring  the 
missingness  process  leads  to  the  correct  inferences  about  0,  since 
(7)  and  (8)  don't  require  the  specification  of  g,  h,  or  t,  and 
standard  computing  algorithms  can  be  used.  Depending  on  why  the 
missing  responses  were  missing,  however,  these  procedures  need  not 
lead  to  the  correct  inferences.  Rubin  (1976,  1987)  specifies 
conditions  under  which  a  missingness  process  can  be  ignored  under 
sampling  distribution,  direct  likelihood,  and  Bayesian  inference. 
They  involve  the  concepts  missing  at  random,  missing  completely  at 
random,  and  distinctness  of  parameters: 

Definition  1:  Missing  responses  are  missing  at  random  (MAR)  if 

for  each  value  of  6  and  for  all  fixed  values  m  and  u  ,  , 

obs 

g,(m|u  .  ,u  ,  )  takes  the  same  value  for  all  u  .  .  (This 

b<j>  '  mis’  obs  mis 

definition  of  MAR  applies  to  the  missingness  process  in  general, 
as  in  Rubin,  1987,  rather  than  a  specific  value  of  the  missingness 
variable,  as  in  Rubin,  1976.) 

Definition  2:  Missing  responses  are  missing  completely  at  random 

(MCAR)  if  for  each  value  of  0  and  for  each  fixed  value  m,  g  (mlu) 

<P 

takes  the  same  value  for  all  u. 

Definition  3:  The  parameter  6  is  distinct  (D)  from  <j>  if  their 
joint  parameter  space  factors  into  a  0-space  and  a  ^-space,  and 
when  prior  distributions  are  specified  for  0  and  <f> ,  they  are 
independent . 


12 


Taken  together,  MCAR  and  D  imply  that  the  values  of  both  the 
observed  and  the  missing  responses  are  independent  of  the  pattern 
of  missingness.  MAR  and  D  together  imply  that  the  values  of  the 
missing  responses  are  independent  of  the  pattern  of  missingness, 
conditional  on  the  values  of  the  observed  responses.  MCAR  implies 
MAR. 


Example  (continued) 

In  order  to  satisfy  MCAR,  it  must  be  that  for  each  value  of  tj> 

and  any  value  of  m,  g  (m|u)  takes  the  same  value  for  all  u.  In 

<P 

our  two-item  example,  however,  g , (M-(l ,0) I U— ( 1 , 1) )«1  while 

g  (M-(l , 0) |u«(0 , 0) )«1-^ .  Except  in  the  trivial  case  that  O ,-{0), 

<P  <p 


MCAR  is  not  satisfied. 

In  order  to  satisfy  MAR,  it  must  be  that  for  each  value  of  <f> 

and  any  fixed  values  of  m  and  u  ,  ,  g,(m|u  ,  , u  ,  )  takes  the  same 

obs  <p  '  mis  obs 

value  for  all  values  of  u  .  .  This  condition  is  satisfied 

mis 

trivially  whenever  m-(l,l),  since  there  are  no  missing 
observations.  It  is  also  satisfied  trivially  in  our  example  when 
m-(0,l)  or  m— (0,0),  since  these  missingness  patterns  have 
probability  zero  for  all  u.  The  following  equalities  for  m-(l,0) 
complete  the  verification  of  MAR: 

g  (M-(1,0)|U-(1,0))  =  g  (M-(1,0)|U-(1,1))  -  1 


g  (M-(l,0)|u-(0,0))  -  g  (M=(1,0)|U-(0,1))  -  W  ■ 


# 


We  are  now  in  a  position  to  summarize  Rubin's  conclusions 
regarding  direct  likelihood  and  Bayesian  inference.  First,  a 
more  easily  verified  sufficient  condition: 


o  When  making  direct-likelihood  or  Bayesian  inferences  about 
0  ,  it  is  appropriate  to  ignore  the  process  that  causes 
missing  data  if  missing  data  are  missing  at  random  and  the 
parameter  of  the  missing  data  process  is  "distinct"  from  9 
(Rubin,  1976,  p.  581) 


When  MAR  is 
brought  out 
to  one.  If 
L( 0 , 0 | v)  on 


satisfied,  z  does  not  depend  on  u  .  and  can  be 
of  the  integral  in  (4) ,  which  then  simply  integrates 
D  is  satisfied  as  well,  the  only  dependence  of 
0  is  through  L*(0|uobs)  [-  f,(uobs)]  • 


Under  weaker  conditions  for  ignorability ,  the  integral  need 
not  drop  out  as  it  does  under  MAR,  but  its  value  does  not  depend 
on  0.  Necessary  and  sufficient  conditions  are  as  follows: 


"k 

o  Suppose  L  (0|u  ) 

for  0  ignoring  the 
correct  for  all  <j> 
(b)  for  each  <t>  t  Cl 


>  0  for  all  0  €  . 

process  that  causes 

0 , ,  if  and  only  if 
9 


All  likelihood  ratios 
missing  data  are 
(a)  =*  nff  x  and 


[g,(ml 


mis 


Umis ’ Uobs 


)lra’Uobs’*’*] 


(9) 


takes  the  same  positive  value  for  all  0.  (From  Rubin,  1976, 
Theorem  7.2.) 
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o  The  posterior  distribution  of  9  ignoring  the  process  that 

causes  missing  data  equals  the  correct  posterior  distribution 
of  9  if  and  only  if 


E  ■  [g  ,(m  u  .  ,u  .) |m,u 

4>,u  .  1  mis  obs  1 

mis 


obs 


J) 


or 


JTg.(m|u  .  ,u  )  p(u  .  Ifl)  p(<^|0)  d<£du  . 
JJb<j>  •  mis  obs  r  mis'  r  1  mis 


takes  a  constant  positive  value.  (From  Rubin,  1976, 
Theorem  8.2.) 


Example  (continued) 

Equation  6  gives  the  complete-data  likelihood  for  the 
observed  data  v  -  (0,*),  namely  L(0,^|V  -  (0,*)).  When  does  the 
psuedo- likelihood  L  (tf|u^-0)  yield  the  same  direct  likelihood 
inferences  about  9?  For  this  to  happen,  it  must  first  hold  that 
the  9  and  <f>  sample  spaces  are  distinct;  it  cannot  be,  for 
instance,  that  the  observed  pattern  of  missingness  could  occur  for 
some  values  of  9  but  not  for  others.  Second,  the  following  term 
that  appears  in  (6)  must  be  constant  for  all  values  of  9: 

W0)  g^[M-(l,0)|U-(0,0)]  +  f/V1)  g^[M-(l,0)|U-(0,l)]  . 
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MAR  would  mean  that  g  .  (M=( 1 , 0) |u=u]  is  constant  for  all  u,  in 

<p 

which  case  the  expression  simplifies  to 


(U2"°)  +  W1)]  ]  ; 


then,  since  the  sum  in  brackets  is  one,  simply  to  the  constant 
value  g  [M-(1,0)].  When  this  happens,  the  sufficient  condition 

<p 

for  ignorability  is  satisfied.  But  even  if  g , f M— ( 1 , 0) lu-rn]  is  not 

<P 

constant  over  u,  the  entire  expression  can  be  constant  for  8  if 
the  variations  in  f  and  g  over  8  cancel  each  other  out.  For 


example,  it  could  be  that  <f>  -  8  and,  for  u2  -  0,1, 


g^[M-(l,0)|U-(0,u2)]  -  [fe(U2«u2)] 


-1 


As  we  shall  see  in  the  case  of  intentional  omissions,  such 
constraints  are  not  generally  plausible  in  the  context  of  IRT.  # 


When  ignorability  under  direct  likelihood  inference  holds  for 

a  given  missingness  process --as  occurs  when  MAR  is  satisfied- -  the 

correct  value  of  8  is  identified  as  the  MLE.  The  usual  sampling- 

A  2 

distribution  interpretation  of  8  and  a  may  or  may  not  be 
justified.  (Recall  that  if  the  sampling  interpretation  is  to  be 
justified  at  all,  it  will  be  with  respect  to  repeated  response 

~  A 

sampling  with  m  fixed  at  m.  The  variance  of  8,  for  example,  may 
be  quite  different  in  this  frame  of  reference  from  its  variance 
under  repeated  samples  of  (u  ,  ,m).) 


First,  a  sufficient  condition. 


o  When  making  sampling-distribution  inferences  about  statistics 
T(v),  it  is  appropriate  to  ignore  the  process  that  causes 
missing  data  if  missing  data  are  missing  completely  at  random 
and  the  parameter  of  the  missing  data  process  is  "distinct" 
from  6.  (Little  and  Rubin,  1987,  p.  14) 


Under  these  conditions,  p(U-u,M-m|  8 ,  -  f  (u)  g  (m)  with  8  and  <f> 

U  <p 

distinct,  and  v  may  be  thought  of  as  the  outcome  of  a  two -stage 
experiment:  4>  determines  m  in  the  first  stage  and  8  determines 
Uobs  t*ie  secon<*-  An  experimenter  looking  at  the  results  of  the 
second  stage  has  the  same  information  about  8  as  an  experimenter 
who  has  performed  that  latter  experiment  only  with  the  value  m 
predetermined. 

A  necessary  and  sufficient  condition  for  ignorability  for 
sampling  inferences  about  a  generic  statistic  T(v) ,  based  on 
Rubin's  (1976)  Theorem  6.2,  is  as  follows: 


o  The  sampling  distribution  of  T(v)  under  f calculated  by 
ignoring  the  process  that  causes  missing  data  equals  the 
correct  conditional  sampling  distribution  of  T(v)  given 
m  under  under  f^  and  g  if  and  only  if  for  each  fixed 
value  of  m,  v 
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Eu  lg,(m|u) |ra,uobs,5 ,<j>]  -  Eu[g  (m|u)|m,*,*]  >  0  . 

Equivalently,  the  probability  of  each  missingness  pattern  must  not 

depend  on  the  values  of  the  responses  that  are  observed;  for  each 

fixed  m  and  any  u'.  and  u' '  ,  it  must  be  true  that 

obs  obs 


Pr (M=m | U 


,  =u' 
obs  obs 


pr(«-»|Uobs-u-!i.»,*) 


This  condition  is  implied  by  Rubin's  (1976)  slightly  stronger 

"observed  at  random."  Unless  it  holds,  (u  ,  ,m)  does  not  admit  to 

obs 

a  decomposition  into  a  sequence  of  independent  experiments  because 
the  value  of  plays  a  role  in  determining  M,  and  the 

conditional  frame  of  reference  is  not  appropriate. 

Inferences  about  Examinee  Ability 
The  following  sections  address  in  turn  the  common  types  of 
missingness  in  IRT  that  were  listed  in  the  introduction,  in  the 
problem  of  drawing  inferences  about  6  when  0  is  known.  In  each 
case,  we  consider  whether  the  conditions  for  ignoring  missingness 
are  plausible,  and,  when  they  are  not,  discuss  how  the 
missingness  process  might  be  modeled  so  that  inferences  can  be 
drawn . 

Case  1:  Alternate  test  forms 

By  "alternate  test  forms,"  we  mean  tests  whose  items  all  fit 
the  same  IRT  model,  and  which  provide  information  sufficiently 
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similar  that  the  test  administrator  is  indifferent  as  to  which 


form  any  particular  examinee  is  presented.  The  form  an  examinee 
receives  will  depend  a  random  process  such  as  a  coin  flip  or  a 
form-spiralling  scheme.  The  common  practice  in  IRT  applications 
with  alternate  test  forms  is  to  base  inferences  about  8  on 

L*<#Kbs>- 

The  use  of  K  alternate  test  forms  implies  that  only  K 
missingness  patterns,  m^,...,  m^, . . .m^,  can  occur,  where  all  the 
elements  of  m^  are  zero  except  those  that  correspond  to  the  items 
that  appear  in  Form  k.  Denote  their  respective  probabilities  by 
<f> k  -  P(M-m^) .  Assuming  the  IRT  model  means  that  f^(U)  is  as  given 
in  (1);  that  is,  we  assume  that  item  responses  would  be  governed 
by  8  alone,  regardless  of  which  items  would  be  presented.  Even 
though  the  items  of  only  one  form  will  actually  be  presented,  it 
is  possible  to  express  our  assumptions  about  the  connection 
between  the  (hypothetical)  values  of  complete  response  pattern  and 
the  probability  of  the  missingness  pattern  as  follows: 


g^(n>|u) 


^  ail  u  if  m-m^;  i.e.  ,  Form  k 

0  otherwise . 


Since  the  values  of  g  do  not  depend  on  u,  MCAR,  and  therefore 
MAR,  are  satisfied.  Verifying  D  for  likelihood  inference  requires 
that  all  values  of  8  are  possible  with  all  possible  values  of  <)>', 
they  are.  Verifying  D  for  Bayesian  inference  requires  that  prior 
beliefs  about  6  and  <f>  be  independent;  this  is  eminently  reasonable 


as  well.  Having  satisfied  the  sufficient  conditions  MCAR  and  D, 
we  conclude  that  the  missingness  caused  by  the  random 
administration  of  alternate  test  forms  is  ignorable  under  direct 
likelihood  and  Bayesian  inference,  and  under  sampling  distribution 

A 

interpretations  of  9.  Common  practice  is  therefore  justified. 

Case  2:  Targeted  Testing 

Targeted  testing  also  involves  multiple  test  forms,  but  ones 
in  which  the  distributions  of  item  difficulty  differ  from  form  to 
form.  Exploiting  the  fact  that  estimates  of  6  are  more  precise 
when  an  examinee  is  administered  items  with  difficulties  near  6, 
targeted  testing  uses  background  information  y  about  an  examinee 
to  select  a  test  form  that  will  probably  be  more  informative  about 
him  than  other  forms .  For  example,  an  easy  form  and  a  hard  form 
might  be  constructed  from  a  set  of  n  items  calibrated  together 
under  the  same  IRT  model,  then  the  easy  form  could  be  given  to 
first  graders  and  the  hard  form  to  second  graders. 

As  in  Case  1,  the  existence  of  K  forms  implies  that  only  K 
patterns  of  M,  namely  m^ ,  ...,  m^,  can  be  realized.  The 
parameter  of  the  missingness  process  has  values  ^(y) >  which 
indicate  the  probability  that  an  examinee  with  background 
variable  y  will  be  administered  Form  k.  For  at  least  one  k  and 
two  values  y'  and  y'  '  ,  ^(y')  *  this  happens  when  p(0\y') 

*  p(0 |y* ' ) ,  and  the  difficulty  of  Form  k  is  better  suited  to  the 
typical  examinee  with  one  value  of  y  than  the  other.  If  we  denote 
the  easy  and  hard  forms  in  the  two -form  example  mentioned  above  as 
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1  and  2,  and  let  y  denote  grade  level  (1  or  2),  then 
g^(M=m^|U-u,y)  =  ^(y)  for  all  u,  with 


^k(y)  =  p(M=i\|y) 


1  if  y  -  1  and  k  -  1 
0  if  y  —  1  and  k  —  2 
0  if  y  -  2  and  k  -  1 
1  if  y  -  2  and  k  =  2 


Because  g  does  not  depend  on  u,  MCAR  is  satisfied.  Assuming 
that  all  values  of  6  are  possible  at  all  values  of  y--even  if 
they  are  more  likely  for  some  values  of  y  than  others-- 
distinctness  as  required  for  direct  likelihood  inference  is  also 
satisfied.  The  values  of  maximum  likelihood  estimates  of  9  from 

*  I 

L  (^luQbs)  are  therefore  the  correct  values  under  targeted 
testing.  This  is  all  that  matters  for  direct  likelihood 
interpretation  of  the  MLE.  Sampling-distribution  interpretations 
are  also  appropriate,  with  respect  to  repeated  administrations  of 
the  form  that  actually  was  administered. 

Distinctness  as  required  for  Bayesian  inference  is  not 
satisfied.  Prior  beliefs  about  9  and  <f>  are  associated  through  y, 
so  p (6 ,4>)  *  p(9)  p(4>)  ■  Intuitively,  knowing  which  form  an 
examinee  was  administered  under  targeted  testing  is  a  source  of 
information  about  9  since  form  selection  depends  on  prior 
knowledge  about  9  through  y.  This  knowledge  must  be  taken  into 
account  in  Bayesian  inference.  It  is  true,  however,  that  p(0,<^|y) 
—  p(tf|y)  p(<^|y)  .  Bayes-distinctness  is  satisfied  conditional  on 
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y,  and  the  missingness  process  can  be  ignored  conditional  on  v. 
Thus,  the  correct  Bayesian  inferences  under  targeted  testing  are 
obtained  with  L  (tf|u  p(#|y),  but  generally  not  with  L  (^|uo^s) 
P(0). 


Case  3:  Adaptive  Testing 

As  mentioned  for  Case  2,  IRT  measurement  can  be  made  more 
efficient  by  presenting  an  examinee  with  items  that  are 
informative  in  the  neighborhood  of  his  6.  Adaptive  testing  uses 
information  from  an  examinee's  preceding  responses,  and  possibly 
from  from  his  background  variables  y  as  well,  to  select  each  next 
item  to  administer.  As  responses  accumulate,  more  is  known  about 
9  and  successive  item  selections  are  more  accurately  targeted. 

The  datum  observed  in  adaptive  testing  is  a  sequence  of  nobs 

)), 

is  the 


(<n)  ordered  pairs,  S  =  (  ,Uobs(1) ) . ^nobs ’Uobs(nobs) 


>(1)"  '  '  ' 

where  I,  identifies  the  k'th  item  administered  and  U 


k  - - * - obs(k) 

response  to  that  item.  Define  the  partial  response  sequence  as 

the  first  k  ordered  pairs  in  S,  with  the  null  sequence  s^ 

representing  the  status  as  the  test  begins.  Augment  the  set  of  n 

items  with  the  fictitious  Item  0,  the  selection  of  which 

corresponds  to  a  decision  to  terminate  testing.  It  can  be 

written  as  the  nobs+l’st  item  in  the  test,  although  no  response 

is  associated  with  it. 

A  test  administrator  defines  an  adaptive  test  design  by 
specifying  for  all  items  j ,  all  realizable  partial  response 
sequences  s^,  and  all  values  of  y,  the  probabilities  <^(j,s^,y) 
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that  item  j  will  be  selected  as  the  k+l'st  test  item,  after 
observing  the  partial  response  sequence  s^  from  an  examinee  with 
Y-y.  The  dependence  of  item  selection  probabilities  upon  y  allows 
for  a  hard  item  to  be  the  first  one  presented  to  a  high  school 
graduate,  say,  but  an  easy  one  to  be  first  for  a  nongraduate. 

Item  selection  probabilities  in  designs  that  do  not  use  y  can  be 
written  simply  as  ^(j,s^).  We  begin  by  considering  designs  of 
this  latter  type . 

One  example  of  an  adaptive  testing  design  is  Bayesian 

minimum  variance  item  selection  (Owen,  1975) .  In  its  pure  form, 

the  item  that  minimizes  the  expected  posterior  variance  of  8, 

using  the  current  posterior  distribution  p(fl|s^),  is  chosen  as  the 
s  t 

k+1  item  with  probability  one.  To  reduce  the  exposure  of  more 
informative  items,  positive  probabilities  may  instead  be  assigned 
to  several  fairly  informative  items.  Typically,  testing  continues 
until  either  a  desired  level  of  precision  is  reached,  or  a 
predetermined  number  of  items  has  been  administered. 

A  second  example  of  an  adaptive  testing  design  is  the  two- 
item  example  employed  earlier  in  this  paper.  Its  definition  of  g 
corresponds  to  administering  the  first  item  to  all  examinees,  and 
with  probability  the  second  item  to  some  of  the  examinees  who 
answered  the  first  item  incorrectly. 

In  adaptive  testing,  the  probability  of  observing  s  from  an 
examinee  with  ability  8  can  be  built  up  sequentially.  The 
probability  of  selection  for  the  first  item  is  ^(i^.s^).  The 
probability  of  response  t0  item  i^  is  given  by  the  IRT 
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model  as  f^  ^u0bs(l)  ^  con<^^tional  independence,  a  value  that 

does  not  depend  on  the  fact  that  this  item  happened  to  have  been 
presented  first.  The  conditional  probability  of  selection  for  the 
second  item  given  s^  is  -  -a  value  independent  of  &.  The 

probability  of  the  corresponding  response  is  a  va-*-ue 

independent  of  the  identification  of,  and  the  response  to,  the 
first  item.  Continuing  in  this  manner  until  it  is  determined  to 
stop  testing,  with  probability  0(0, s),  we  obtain 


nobs+1  nobs 

P(s|«,  -  n  *<ik.Vl>  n  f,<u  >  . 


The  likelihood  function  induced  by  the  observation  of  s  is  thus 


nobs+1  -  + 

L(.|s)  -  0  L  «|»obs>  . 

k-1 


Observe  that . . . 


1.  s  conveys  the  value  of  m:  iik  —  1  if  i^  -  j  for  some  k, 

1<  k  <  nobs;  otherwise,  m.  =0. 

_  _  ,  j 


2.  s  conveys  the  value  of  u  ^  ,  namely  the  responses  to  the 
items  administered  during  the  course  of  the  test. 
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3. 


L  factors  into  two  products,  the  first  of  which  depends  on 

★  | 

4> ,  u  >  anc*  m>  anc*  t^e  second  °f  which- -namely  L  C^lu0^s) 
--depends  on  6  and  u  ^  . 

4.  If  s'  and  s''  imply  the  same  m  and  u  then  L(fl|s')  « 

L( 9 | s  "  ) . 

Points  3  and  4  justify  the  use  of  L  (^lu0^s)  for  direct 
likelihood  inference.  It  may  be  instructive  nonetheless  to  verify 
the  satisfaction  of  MAR.  Now  P(M-m,U-u),  or  the  probability  of 
the  hypothetical  complete  observation  (m,u),  is  the  probability  of 
observing  a  response  sequence  s  that  yields  the  targeted  m  and 
Uobs ’  c^mes  the  Pr°bability  that  the  unobserved  responses  umis 
also  take  the  targeted  values.  Defining  T  *  (s:  M-m  n  ) 

as  the  set  of  response  sequences  that  present  the  targeted  items 
and  have  the  targeted  responses  to  them,  we  have 


P(M=m,U=u)  -  P(seT)  P(U  =u  .  ) 

mis  mis 


nobs+1 


i  s  n  *(ik.Vl)  n  £,(uob  ><  n  £,(«„  )i 

T  k  obs  mis 


nobs+1 


n 


=  {  Z  n  *(i.,s  ,))  n  f  (u  ) 
Ik  1  j-1  J 
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nobs+1 

-  (  s  n  4>( i.,s.  .)}  P(u-u)  . 

T  k  K  K 


But  then 


g  (m|u)  -  P(M-m,U-u)/P(U-u) 


nobs+1 

2  11  ^(ik'Sk  1>  ’ 

T  k  i 


a  value  that  does  not  depend  on  um^s.  as  required  for  MAE..  This 
argument  also  holds  when  ^  depends  on  y.  (QED) 

MAR  and  distinct  parameter  spaces  are  sufficient  for 
ignorability  of  the  adaptive- testing  missingness  mechanism  under 
direct  likelihood  inference.  Ignorability  holds  under  Bayesian 
inference  if,  in  addition,  the  prior  distributions  for  6  and  4>  are 
independent.  As  with  targeted  testing,  this  latter  condition 
fails  if  for  some  y'  and  y' ’  for  which  p(0|y')  *  p(tf|y''),  there 
exist  j  and  s  such  that  flKj.s.y')  *  <t(j,s,y'').  When  this  is  so, 

Bayesian  inference  demands  the  use  of  p(0|y)  rather  than  p(0)  in 

★  ■ 

conjunction  with  L  (^lu0^s)- 

Even  though  ignorability  under  direct  likelihood  inference 
★ 

means  that  L  yields  the  correct  maximizing  value  from  a  given 

A 

observation,  sampling-distribution  interpretation  of  the  MLE  6  is 
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not  justified  in  general.  To  see  this,  recall  that  the  necessary 
condition  for  sampling-distribution  ignorability  requires  that  for 


each  fixed  m  and  any 


and  u 


t  t 

obs  ’ 


Pr(M-m|U 


obs~Uobs’9 


Pr(M-ra|U 


Obs^obs'6^ 


This  would  require  that  the  probability  of  any  given  missingness 
pattern  be  the  same  no  matter  what  values  the  responses  took.  But 
since  by  definition  adaptive  tests  produce  missingness  patterns  as 
a  function  of  the  response  values  that  are  observed,  only  a 
degenerate  adaptive  testing  scheme  could  satisfy  this  condition. 

Concluding  that  the  item  selection  mechanism  is  not  ignorable 
for  sampling  distribution  inference  means  that  the  correct 

A 

sampling  distribution  for  8  must  be  calculated  with  respect  to 
repeated  administrations  of  the  entire  adaptive  test.  While 
general  theory  does  not  relate  its  variance  in  this  frame  of 
reference  to  the  second  derivative  of  L  ,  the  latter  may  be  a 
reasonable  approximation  of  the  former  under  particular  adaptive 
test  designs.  Whether  this  is  so  must  be  determined  individually 
for  each  adaptive  test  design,  analytically  in  simple  cases  but  by 
simulation  in  more  realistic  cases. 

Case  4:  Not-Reached  Items 

IRT  is  intended  for  "power"  tests,  or  those  in  which  an 
examinee's  chances  of  responding  correctly  would  not  differ 
appreciably  if  the  time  limit  were  more  generous.  Time  limits  are 
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typically  chosen  to  allow  most  examinees  to  respond  to  all  items, 
but  a  few  examinees  won't  have  time  to  answer  all  of  them.  This 
section  concerns  the  items  that  an  examinee  does  not  reach.  It 
assumes  the  examinee  has  not  interacted  with  the  item--e.g.,  he 
has  not  seen  what  the  items  at  the  end  of  the  test  ask,  and 
decided  to  work  instead  on  the  ones  he  has  seen  at  the  beginning 
of  the  test. 

It  is  common  practice  to  identify  not-reached  items  by 
working  from  the  end  of  an  examinee's  response  string  toward  the 
beginning,  taking  unanswered  items  as  not-reached  until  an  answer 
is  encountered.  Unanswered  items  preceding  this  last  answered 
item  are  taken  as  intentional  omissions,  and  will  be  considered 
in  the  next  section.  Concentrating  on  nonresponse  due  to  not- 
reached  only,  and  limiting  our  attention  to  examinees  who  have 
reached  at  least  the  first  the  item,  we  must  address  n  patterns  of 
missingness:  for  2  =  0,...,n-l,  let  m^  denote  the  string  of  n- 2 
l's  followed  by  2  0's.  That  is,  m^  is  the  missingness  pattern  of 
an  examinee  that  has  not  reached  the  last  2  items. 

Checking  ignorabilitv.  We  continue  to  assume  that  a  common 
IRT  model  holds  for  the  responses  of  items  reached,  u  ^ ,  and 
not-reached,  u^.^.  This  assumption  is  crucial  for  applying  IRT 
models  to  data  with  not-reached  items,  and  two  ways  of  checking  it 
will  be  discussed  at  the  end  of  the  section.  When  it  does  hold, 
the  missingness  process  is  characterized  by  the  examinee  speed 
parameter  4>  -  •  •  •  ,4>n  of  a  multinomial  variable,  where  <j> ^ 

is  the  probability  that  missingness  pattern  m^  will  be  observed-- 


i.e.,  that  the  last  l  items  will  not  be  reached.  Under  this 
formulation,  the  probability  of  the  complete  observation  (m,u)  is 
obtained  as 

n 

p(m,u|  8  ,<f>)  -  p(mj^)  n  p(u  |0)  ; 

j  J 

MCAR  (and  therefore  MAR)  holds.  The  probability  of  v  is 

p(v\6,<t>)  -  p(m| *)  P(^obsl«)  f  PC^.Jtf)  dumis 

-  p(m|^)  P(uobsl^>  •  (12) 

If,  in  addition  to  MCAR,  all  values  of  8  are  possible  at  all 
values  of  ^--even  if  some  are  more  likely  than  others- -the 
not-reached  missingness  process  is  ignorable  with  respect  to 
direct  likelihood  inference.  That  is,  direct  likelihood  inference 
about  8  in  the  presence  of  not-reached  items  can  be  based  on 

*  i  * 

L  (®luobs)-  Sampling-distribution  inferences  about  8  from  8  are 
also  appropriate.  They  pertain  to  repeated  sampling  of  responses 
to  the  items  that  were  reached,  and  enjoy  the  asymptotic  sampling 
properties  of  MLEs  if  the  number  of  items  reached  is  large. 

For  ignorability  to  hold  under  Bayesian  inference,  it  is 
necessary  in  addition  to  MAR  that  p(0,^)  -  p  (8)  p(<f>)  ;  that  is, 
that  "speed"  and  "ability"  are  independent.  Empirical  evidence 
suggests  that  this  is  not  generally  true.  Van  den  Wollenberg 
(1979),  for  example,  reports  significant  positive  correlations 
between  percent-correct  scores  on  the  first  eleven  items  (which 
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were  reached  by  all  examinees)  and  the  total  number  of  items 
reached,  in  four  of  six  intelligence  tests  in  the  ISI  battery 
(Snijders,  Souren,  and  Welten,  1963).  Bayesian  inference  about 
8  would  take  this  relationship  into  account  by  using  the  correct 
posterior  distribution 

P(*|v)  oc  f  L(0  ,  ^  |  v)  p<* ,*)  d* 

-  /  L*(fi|uQbs)  L(*|m)p(*|f)  d 8  p (8) 

-  L*(«|uobg)  P<«)  f  L(*|m)  p(*| 8)  d 8 

“  L*(<,luobs)  • 


Checking  the  IRT  model.  Verifying  MCAR  for  not-reached  items 
required  assuming  that  the  responses  that  would  have  been 
observed,  had  those  items  been  reached,  follow  the  same  IRT  model 
as  those  that  were  reached.  We  now  describe  two  ways  of  checking 
this  assumption,  one  using  only  the  response  data  v  that  are 
normally  observed,  the  other  requiring  the  researcher  to  discover 
not-reached  responses  in  a  supplemental  data- gathering  effort. 
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A  necessary  condition  for  an  IRT  model  to  hold  in  the 


presence  of  not-reached  items,  is  that  the  same  IRT  model  hold 
for  reached  items  among  examinees  who  have  reached  different 
numbers  of  items.  Let  be  the  observed  responses  to  items 

that  are  reached  in  a  sample  of  N  examinees,  and  let  nobs^  be  the 
number  of  items  examinee  i  reaches.  Let  0  ^  be  the  parameter(s)  of 
item  j .  The  marginal  probability  of  u  ^  under  the  hypothesis 
that  item  parameters  are  invariant  over  groups  of  examinees  with 
different  missingness  patterns  is 

nobs . 

PA(Uobs)  ~  n  f  n  1  P<u  )  p<*|rn.)  dff  .  (13) 

i  j-1  J  J 


Viewing  (13)  as  a  likelihood  function  and  maximizing  with  respect 

to  0,  ....  ,0  yields  the  value  L. . 

1  n  A 

An  alternative  hypothesis  is  that  item  parameters  vary  over 

not-reached  groups.  We  can  estimate  n-j+1  different  item 

parameters  for  item  j,  where  0..  applies  to  those  examinees  who 

J  * 

have  reached  n-i  items.  For  example,  Item  3  will  have  parameters 
for  groups  who  reached  n,  n-1,  ....  4,  and  3  items.  The  marginal 
probability  of  under  this  hypothesis  is 


nobs , 


'n'W  -  7  I  p<uljl*'^j,nobs.)  M  ■ 

1  J”1  1 
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which  leads  to  the  maximizing  likelihood  value  L  .  In  large 

B 

samples,  -2  log(L  /L  )  is  approximately  chi-square  with  n(n  -  l)/2 
degrees  of  freedom  when  ' ae  null  hypothesis  is  true. 

Van  den  Wollenberg  (1979)  provides  empirical  evidence  that 
the  "item  parameter  invariance"  with  respect  to  not-reached  groups 
is  often,  but  not  always,  tenable.  Applying  his  own 
goodness  -  of  -  fit  indices  rather  than  the  likelihood  ratio 
suggested  above,  he  verified  this  type  of  invariance  in  five  of 
the  six  ISI  tests. 

A  second  way  of  studying  the  IRT  assumption  in  the  presence 

of  not-reached  items  begins  by  finding  out  what  the  responses  to 

the  not-reached  items  would  have  been.  This  can  be  accomplished 

with  paper-and-pencil  tests  by  allowing  a  sample  of  examinees  to 

continue  beyond  the  usual  time  limit  until  they  have  answered 

every  item,  but  using  a  different  colored  pencil  after  the  usual 

limit.  Of  the  total  of  n  items,  then,  examinee  i  will  have 

responded  to  the  first  nobs^  under  the  normal  time  limits  and  the 

remaining  nrnis^  =  n  -  nobs^  thereafter.  Under  the  null 

hypothesis  of  an  invariant  IRT  model  across  reached  and  not- 

reached  items,  the  marginal  probability  of  the  completed  response 

matrix  u  =  (u  .  ,u  ,  )  under  the  null  hypothesis  is 

mis  obs  yr 

PC(Uobs)  “  11  J*  11  P<Uii  1 19  ■ 

1  j  J 
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Alternatively,  we  can  fit  an  IRT  model  that  allows  both  the  item 

parameters  and  examinee  parameters  to  differ  before  and  after  the 

time  limit.  Each  item  except  the  first  can  have  two  parameters, 

B.  .  and  8.  ,  ,  whenever  some  examinees  answered  before  the 

j ,mis  J , obs 

time  limit  and  some  answered  it  after;  each  examinee  can  two 
abilities,  0^^  and  0  s>  The  resulting  marginal  probability  is 


nobs . 
l 


pD(Uobs)  “  11  f  f  1  ”  p(uijl*obs’/?j,obs. 

l  J-l  J  i 


)] 


runts . 

l 


In  large  samples,  -2  log(L^/L^)  is  approximately  chi-square  under 

the  null  hypothesis,  with  degrees  of  freedom  equal  to  the  number 

of  items  with  two  parameters  appearing  in  (14)  ,  plus  the  number 

of  additional  parameters  estimated  for  the  examinee  parameter 

distribution  p(0  .  ,8  .  )  over  those  required  for  p(0). 

r  obs  mis  r 

Case  5:  Omitted  Responses 

A  missing  response  is  an  intentional  omission  when  the 
examinee  is  administered  the  item,  has  time  to  appraise  its 
content,  and  decides  for  his  own  reasons  not  to  make  a  response. 
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After  showing  that  such  omissions  can't  generally  be  considered 
ignorable,  we  discuss  a  number  of  ways  to  deal  with  them. 

Omitting  behavior.  Test  scores  T(v)  are  assigned  to 
patterns  of  rights,  wrongs,  and  omits  for  the  purposes  of 
comparing  or  selecting  examinees.  Assuming  that  a  correct 
response  to  an  item  gives  a  higher  value  of  T  than  an  incorrect 
response  and  that  an  examinee  wants  to  obtain  a  high  score,  he 
will  make  responses  he  believes  are  correct.  How  he  will  respond 
to  an  item  about  which  he  is  unsure  depends  at  least  partly  on 
how  the  test  will  be  scored  (Sabers  and  Feldt,  1968). 

Formula  scores,  for  example,  take  the  form 

T(v)  -  R(v)  -  X  W(v)  , 

where  R(v)  and  W(v)  are  counts  of  right  and  wrong  responses  and  X 
is  a  constant  selected  by  the  test  administrator.  Setting  X  -  0 
gives  number-right  scores;  X  -  1  gives  right-minus-wrong  scores; 
for  multiple  choice  items  with  A  alternatives,  X  -  1/(A-1)  gives 
the  familiar  "corrected-for-guessing"  scores.  The  examinee 
maximizes  his  expected  score  by  answering  items  for  which  he 
thinks  his  chances  of  being  correct  are  at  least  X/(l  +  X).  I 
particular,  he  should  answer  every  item  under  number-right 
scoring,  and  those  for  which  he  thinks  his  chances  are  at  least  c 
=  1/A  under  corrected-for-guessing  scoring.  Some  examinees  either 
do  not  use  this  strategy,  or  make  inaccurate  assessments  of  their 
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chances.  Analyzing  responses  to  items  that  examinees  originally 
omitted  under  right-minus -wrong  scoring,  Sherriffs  and  Boomer 
(1954)  did  find  about  half,  or  X/(l  -  X),  the  omitted  responses 
would  have  been  correct  among  examinees  who  scored  low  on  a 
risk-aversion  scale,  but  nearly  two-thirds  would  have  been  correct 
among  examinees  with  high  risk-aversion  scores. 

The  examinee's  perceived  probabilities  of  correct  response 
must  be  distinguished  from  the  probabilities  of  the  IRT  model. 

IRT  gives  the  proportion  of  correct  response  to  Item  j  from 
examinees  with  ability  9  ,  but  each  of  these  examinees  may  have  a 
different  idea  of  his  own  chances.  They  may  differ  in  the 
accuracy  of  their  estimates  and  their  confidence  about  them,  and 
their  perceived  probabilities  need  not  average  to  the  IRT 
probability.  Observing  whether  an  examinee  omits  an  item  merely 
tells  us  something  about  what  he  thinks  u^  would  be. 

Are  omits  jgnorable?  To  see  that  the  assumptions  needed  for 

ignorability  are  not  generally  plausible,  we  examinee  the  case  in 

which  n  =  1;  i.e.,  a  single  item.  MAR  simplifies  to  g  (mlu-=0)  - 

<P 

g  (m|U=l) ,  meaning  that  an  omit  is  just  as  likely  if  the  response 
<p 

would  have  been  correct  as  if  it  would  have  been  incorrect.  But 
since  examinees  tend  to  answer  items  they  feel  are  correct,  MAR 
implies  the  unappealing  assumption  that  their  perceptions  of 
correctness  are  independent  of  actual  correctness. 

MAR  (along  with  D)  is  merely  sufficient  for  ignorability, 
however,  and  ignorability  can  hold  when  MAR  does  not.  For  a 
single  item,  the  necessary  condition  for  ignorability  under 
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likelihood  inference  given  in  (9)  requires  that  for  each  value  of 
<f> ,  the  expression 

g^(M=0|U=0)  f ^ (U=0)  +  g  (M-0|U-1)  f^(U-l)  (15) 

take  the  same  value  for  all  8  .  This  is  the  just  the  probability 
that  the  item  will  be  omitted.  That  its  value  remains  constant  as 
6  increases  without  limit  flouts  intuition,  since  we'd  expect 
examinees  whose  high  abilities  virtually  assure  a  correct  response 
to  be  aware  of  their  high  chances,  and  respond  rather  than  omit. 
This  conjecture  is  borne  out  in  empirical  studies  such  as 
Stocking,  Eignor,  and  Cook  (1988)  that  show  markedly  lower  rates 
of  omission  from  examinees  with  high  (corrected- for-guessing) 
scores  than  from  examinees  with  low  scores. 

Since  ignorability  is  not  satisfied  for  direct  likelihood 

■jJr 

inference,  L  does  not  generally  yield  the  correct  MLE,  and 
sampling  distribution  inferences  based  on  the  resulting  value  are 
inappropriate.  The  requirements  for  ignorability  under  Bayesian 
inference  are  the  same  as  those  for  direct  likelihood  inference 
except  that  they  must  apply  when  averaged  over  4>  rather  than  for 
each  particular  value;  ignorability  is  thus  implausible  there  too. 

Lord  (1974)  argues  against  ignoring  omits  under  maximum 
likelihood  scoring,  saying  that  the  examinee  who  knew  we  planned 

"k 

to  use  the  MLE  from  L  (fllu  ,  )  as  a  score  would  omit  all  items 

1  obs 

except  those  for  which  he  was  certain  his  response  would  be 


correct.  This  plausible  argument  also  presumes  a  relationship 
between  actual  and  examinee -perceived  item  correctness. 

Filling  in  the  blanks.  Lord  (1974)  suggested  that  omits  on 
multiple-choice  items  under  guessing-corrected  scoring  can  be 
handled  with  standard  IRT  estimation  routines  if  they  are  treated 
as  as  fractionally  correct,  with  value  c.  He  assumed  "rational" 
omitting  behavior:  examinees  omit  items  only  if  their  chances  of 
responding  correctly  would  have  been  c,  so  that  h^(lh=l |ff.=0)  ~  c 
for  all  items  and  all  8  .  Omitting  decisions  are  also  assumed  to 
be  independent  from  one  item  to  the  next,  given  6  and  <j> .  In  a 
natural  extension  of  conditional  independence  of  item  responses 
given  9,  we  assume  "extended  local  independence,"  or  conditional 
independence  of  item  responses  and  missingness  given  8  and  <f>\ 

P(U-u,M-m|0  ,<j>)  =  n  P(U.-u.  ,M.-m.  |  9  ,<j>)  . 

j  J  J  J  J 

Under  these  assumptions,  the  complete-data  likelihood  takes 
the  following  form: 

-  -  n  -  - 

L(0  ,<0|u,m)  -  n  p(u  ,m  |  8  ,<j>) 
j  J  J 

n  -  -  - 

-  n  f5(Uj)g(nu  |uj  ,6  ,<f>) 
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U  X*  .  XU,  ll  * 

-  n  P  (tf)  J  Q  (9)  J  n  g(m  |u  ,8,4>) 

j  J  J  j  J  J 

[where  P AB)  -  f.(U  -1)  and  Q  (0)  =  1  -  P.(0)) 

J  9  J  J  J 

-  Lu<*|u>  Lm|u<*.«H<ralu))  • 


The  complete -data  likelihood  thus  factors  into  two  terms,  with 
being  the  IRT-based  probability  of  item  responses  and  the 

probability  of  the  missingness  pattern  given  the  response 
pattern.  Both  depend  on  9.  Were  u  and  m  both  fully  observed, 
the  usual  MLE  based  on  would  be  a  conditional  MLE,  foregoing 
the  additional  information  conveyed  by  | ^  but  avoiding  the 
nuisance  parameter  <f> .  One  would  proceed  by  finding  the  maximizing 
value  of  the  log  likelihood 


n  - 

*u  -  2  Uj  log  Pj(«)  +  (1  -  Uj)  log  Qj(0) 


The  same  conditional -estimation  strategy  can  be  applied  to 

the  observed  data  v  -  (u0^s>m)»  by  maximizing  the  conditional 

expectation  of  L  ,  or  E[L  (0|u  .  ,u  ,  ) I (u  ,  ,m)l.  Finding  the 

u  u  '  mis  obs  '  obs 

maximizing  6  for  E(L^)  by  Dempster,  Laird,  and  Rubin's  (1977)  EM 
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algorithm  requires  finding  the  maximizing  0  for  the  expectation  of 
•*u.  given  uQbs,  m,  and  a  provisional  estimate  0°;  that  is,  of 


F(0|0°)  -  f  £  (8 |u  ,u  ,  )  p(u  .  j u  ,m, 0  -  0°)  du  . 

11  mi  C  One  4  ml  C  1  rvh  C  mi  i 


S  u  log  P.(0)  +  (1  -  u  )  log  Q . ( 0 ) 
obs  J  J  J  J 


+  J[  Z  u.  log  Pj(0)  +  (l-Uj)Qj  (0)]  P(umisluobs-m-6,_e°)  du, 

(16) 

But  under  Lord's  assumptions, 


mis 


p (u  .  j u  ,  ,m,i 
mis'  obs 


»°)  -  n  h  (u  |m  -  0) 
mis  d 


u.  1-u. 

-  n  c  J(1  -  c)  J 
mis 


a  value  that  doesn't  depend  on  <f>  or  0*^.  Substituting  this 
expression  into  the  second  term  of  (16),  the  integral  simplifies 
to 


J 


a 
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F(*|fl  )  -  S  u.  log  P  (S)  +  (1  -  u.)  log  Q.(ff) 


obs 


+  E  c  log  P  (*)  +  (1  -  c)  log  Q  (6) 
mis  -1 


(17) 


This  is  equivalent  to  the  log  of  Lord's  Equation  4,  the 
psuedo-likelihood  obtained  by  using  u^  -  c  in  the  complete-data 
likelihood  whenever  nu  —  0.  Equation  (17)  does  not  depend  on  6^, 
so  the  EM  algorithm  comprises  only  a  single  cycle.  Maxima  of  (17) 
are  maxima  of  E(Lu) .  A  global  maximum  is  assured  if  the 
complete-data  probability  belongs  to  the  exponential  family,  as  is 
the  case  with  the  Rasch  model. 

Lord  (1974)  points  out  that  the  criterion  function  obtained 
by  replacing  omits  with  fractionally-correct  responses  is  not  the 
likelihood  function  induced  by  the  observation.  We  have  shown, 
however,  that  the  resulting  estimate  of  6  maximizes  what  might  be 
called  a  "marginal  conditional"  likelihood  function,  allowing  one 
to  apply  standard  results  from  the  theory  of  maximum  likelihood 
estimation,  such  as  consistency- - in  this  context,  as  the  number  of 
items  not  omitted  increases. 

The  foregoing  analysis  yields  insight  into  other  treatments 
of  omits  that  impute  values  for  .  Supplying  random  responses 

that  are  correct  with  probability  c  provides  a  crude  numerical 
evaluation  of  (16),  leading  to  a  maximizing  value  whose 
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expectation  is  the  value  obtained  when  the  integration  is  carried 
out  in  closed  form  as  in  (17).  This  practice  is  justified  by  the 
same  assumptions  as  Lord's  (1974)  approach,  but  sacrifices 
accuracy  for  convenience.  Supplying  incorrect  responses  for 
omits  leads  to  a  "marginal  conditional"  MLE  for  8  under  the 
assumption  that  responses  to  omitted  items  would  surely  have  been 
incorrect.  This  may  be  reasonable  for  open-ended  items,  but  it  is 
not  plausible  for  multiple -choice  items  for  which  even  the  least 
able  examinees  have  nontrivial  probabilities  of  success.  In  these 
cases,  supplying  incorrect  responses  for  omits  would  bias 
estimates  of  8  downward. 

Lord  addressed  "rational"  omitting  behavior,  in  that  the 
expectation  of  correctness  for  an  omitted  response  is  always  c, 
the  value  associated  with  the  optimal  omitting  strategy.  As  we 
have  noted  however,  studies  of  responses  to  items  originally 
omitted  show  that  not  all  examinees  behave  in  this  manner.  The 
tendency  to  omit  when  probabilities  of  success  may  be  higher  than 
c  can  be  associated  with  personality  characteristics,  demographic 
variables,  and  level  of  ability.  This  approach  biases  estimates 
of  8  downward  for  risk-aversive  examinees.  We  now  discuss  how 
such  dependencies  can  be  taken  into  account,  although  it  is  by  no 
means  certain  that  this  should  be  done;  to  do  so  effectively 
adjusts  scores  upward  or  downward  in  accordance  with  examinee 
background  characteristics,  which  may  be  objectionable  on  the 
grounds  of  fairness.  Assuming  rational  omitting  behavior  in 
scoring  rules,  and  making  the  rules  and  optimal  strategies  as 


clear  as  possible  to  examinees,  may  be  preferable  when  test 
scores  are  used  to  make  sensitive  placement  or  selection 
decisions . 

Modeling  empirical  rather  than  ideal  omitting  behavior 
requires  a  study  like  that  of  Sherriffs  and  Boomer's  (1954), 
where  examinees  are  first  administered  a  test  under  standard 


conditions,  then  later  asked  what  their  responses  to  the  items 
they  omitted  would  have  been.  From  these  data  it  is  possible  to 
calculate  proportions  of  omits  that  would  have  been  correct  as  a 
function  of  the  items  and  examinee  characteristics* -possibly 
including  8.  If  S  is  not  included,  empirical  estimates 
h(Uj-l  |Mj-=0,y)  are  employed  in  place  of  c  in  (16).  This  takes 
into  account  possible  differences  in  rates  of  omitted  correct 
response  from  one  item  to  the  next- -some  higher  than  c,  some 
lower- -or  among  examinees  with  different  demographic  or  measured 


psychological  characteristics .  If  6  is  included,  then  estimates 


of  8  employing  h^(U^-  1 1 M j  -0  ,  y ) 

The  values  h  (U.-l|M.-0,y)  . 

8  j  1  j  \ 


must  be  calculated  iterative 
0  replace  c  in  (17)  for  each 


iy. 


missing  response,  and  an  improved  estimate  8  is  obtained  via 


maximum  likelihood.  This  must  then  be  used  to  produce  an  improved 


estimate  of  the  expectation  of  each  missing  response, 

h.(U.“l|M.-0,y)  .  .1.  From  these  and  u  ,  ,  yet  another  estimate 

8  j  1  j  \ obs 

2 

8  will  be  obtained.  The  process  is  repeated  until  convergence 


occurs.  The  original  estimation  of  item  parameters  and  of  the 
functions  h^(Uj-l|Mj-0,y)  requires  similar  modifications  to 
standard  item  parameter  estimation  algorithms.  Additional 
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parameters  for  h^  (lh-1  |Mj-0  ,y)  can  be  estimated  jointly  with 
standard  item  parameters;  a  plausible  implementation  would  make 
the  logits  of  h's  associated  with  each  item  linear  in  9. 


Lord's  (1983)  model  for  omits.  While  Lord’s  (1974) 
treatment  of  omits  as  fractionally  correct  yields  reasonable  and 
statistically  defensible  (conditional)  MLEs  of  9  when  rational 
omitting  behavior  is  assumed,  the  full  likelihood  induced  by  the 
data  was  neither  presented  nor  exploited.  To  accomplish  this 

I 

requires  an  explication  of  the  missingness  process,  in  the  form  of 
a  model  for  the  joint  probability  distribution  of  U  and  M.  Such  a 
model  was  proposed  by  Lord  (1983). 

Lord's  (1983)  model  for  omits  maintains  the  context  of 
guessing-corrected  scoring  of  multiple-choice  items  with  A(«  1/c) 
alternatives,  but  offers  additional  structure  for  the  response 


process.  It  is  first  assumed  that  an  examinee  either  feels  a 

preference  for  one  of  the  alternatives  or  is  totally  undecided 

among  them.  The  proportion  of  examinees  with  ability  9  feeling 

no  preference  on  Item  j  is  Rj(0).  If  a  preference  is  felt,  a 

response  is  made;  of  the  responses  made  by  examinees  with  ability 

* 


9  who  feel  a  preference,  the  proportion  correct  is  P  ^(0).  If  no 
preference  is  felt,  the  examinee  will  either  omit  the  item  with 
probability  w  or  respond  completely  at  random.  Responses  and 
omitting  decisions  are  again  assumed  to  be  independent  from  one 
item  to  the  next,  conditional  on  9  and  w. 
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These  assumptions  imply  that  the  missingness  parameter  ^  is 

(0,u>),  and  lead  to  the  following  forms  for  t  (m.)  and  h  (u.lm.): 

<P  J  s  J  J 


1-m.  m. 

t  (m. )  =  w  R .(0)  3  [1  -  uR . ( 0 ) ]  3 

<t>  J  J  J 


and 

if.m  -  0 

J 

if  m.  -  1, 

J 


h/u.|m.) 


u.  1 - u . 

c  3  (1  -  c)  ^ 


**  ui  **  1_ui 

L  Pj  (*)  J[1  -  Pj  (0)]  J 


where  P^  (0)  =  P(U^-1 | 0 ,m.“l) ,  the  conditional  probability  that  an 
observed  response  will  be  correct,  is  the  sum  of  the  probabilities 
of  responding  correctly  when  a  preference  is  felt  and  guessing 
correctly  when  a  preference  is  not  felt: 


?.  (0)  ~  [1  -  R j ( 0 ) ]  Pj(0)  +  c(l  -  w)R  (*) 


The  joint  likelihood  for  0  and  w  induced  by  v  (i.e., 
is  thus 


u  .  and  m) 
obs 
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1 


n 


L(0,w|v)  -  t^(m)  /  II  (Uj  |  m j  )  du 


mis 


t  (m)  n  h  (u  Im.  -  1)  f  n  h  (u. |m.  -  0)du  . 

<(>  '  ^  1 1  j  J  .  8  j 1  l  mis 

obs  J  J  mis  J  J 


n  1-m. 

n  [«R  («)]  J[i 

j  J 


-  WR4(fl)]  j  n  p**  j(«)Q**(«)  j 


obs 


(18) 


Assuming  the  functions  P  and  R  are  known,  (18)  provides  a  basis 
for  full- information  inference  about  8.  Under  maximum 
likelihood,  the  joint  maximum  for  (8 ,w)  may  be  found  by  standard 
numerical  methods,  and  a  large-sample  variance  estimate  can  be 
based  on  the  inverse  of  the  second  derivative  of  the  log  of  (18) 
with  respect  to  8  and  w.  Under  Bayesian  inference,  the  posterior 
for  8  and  w  is  obtained  by  multiplying  (18)  by  p(0,w)  and 
normalizing;  from  this  point,  one  may  examine  characteristics  of 
the  joint  posterior  for  9  and  u>,  or  integrate  w  out  to  obtain  the 
marginal  posterior  for  9 . 

Lord  suggests  that  this  model  might  be  implemented  by 

★ 

specifying  functional  forms  for  P  and  R,  e.g.,  the  3-parameter 

logistic  IRT  function  for  P  and  the  2 -parameter  logistic  with  a 

negative  slope  for  R.  The  underlying  model  for  the  correctness  of 

item  responses,  observed  or  not,  can  be  written  as  a  function  of 
* 

P  and  R  as  follows; 
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P(U 


P(Uj-l|tf,«,Mj-l)  P(Mj-l|«,u)  +  P(Uj-l|tf,w,M  -0)  P(M  -0|*,w) 


lc(l  -  w)Rj(0)  +  [1  -  Rj(tf)]Pj(«>)  [1  -  wR  (*)]  +  cwRj(9) 


Note  that  this  probability  depends  on  u>  as  well  as  6\  thus,  the 


underlying  model  for  item  responses  U.  is  not  a  standard  IRT  model 


In  a  manner  described  by  Samej iraa  (1979),  a  response  curve  of 
this  form  need  not  be  monotonically  increasing  over  the  range  of 
6.  High-ability  examinees  would  tend  to  feel  preferences  and 
respond  correctly;  moderate  -  ability  examinees  might  tend  to  feel  a 
preference  for  a  clever  distractor  and  answer  incorrectly  at  a 
rate  lower  than  c;  very  low  ability  examinees  would  feel  no 
preference  at  all,  and  answer  correctly  at  a  rate  equal  to  c. 
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Nominal  category  models.  IRT  models  for  multiple-category 
items  have  been  proposed  by  Bock  (1972),  Samejima  (1979),  Sympson 
(1983),  Thissen  and  Steinberg  (1986),  and  others.  These  models 
have  sometimes  been  used  for  data  with  intentional  omissions, 
with  an  omit  treated  as  one  more  possible  response  to  a  multiple- 
choice  item.  Lord  (1983)  expresses  reservations  about  this 
practice , 

"...since  it  treats  probability  of  omitting  as  dependent 
only  on  the  examinee's  ability,  whereas  it  actually 
depends  on  a  dimension  of  temperament.  It  seems  likely 
that  local  unidimensional  independence  may  not  hold." 

(p.  477) 

The  following  analysis  makes  Lord's  concerns  more  explicit. 

The  features  of  the  approach  regarding  omission  are  retained 

when  all  overt  incorrect  responses  are  collapsed  into  a  single 

category.  Recalling  that  the  values  0,  1,  and  *  of  v  stand  for 

observed  wrong,  observed  right,  and  omit,  we  obtain  the  multiple- 

★ 

category  model  probabilities  f  as  follows: 

u 

f*(V.  -  0)  -  P(U.  -  0,M.  -  l|0) 
p  J  J  J 

-  /  f,(U  -  0)  g(Mj  -  1 1 Uj  -  0,9,4,)  P(*|0>  d* 

f*(V.  -  1)  -  P(Uj  -  l.M  -  l|f) 

-  J  ftf(U  -  1)  g(M  -  1|U  -  1 ,9,4,)  p($| 9)  d 4 

and 
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f>.  -  *)  -  P(Uj  -  0 ,Mj  -0)$)  +  P(U  -  l.M  -0| (9) 

-  f  ffi(U  -  1)  g(M  -  0|u  -  IJ.t)  p<*|«)  d 4 

+  f  fg(u  -  0)  g(M  -  o|u  -  o ,M)  P(*|ff)  d 4> 


Under  the  assumption  of  "extended  local  independence,” 


p(U  -  u,M  -  m|0,4) 


n  p(U 

j  J 


u.  ,M. 
j  J 


nij  |  8 ,<f>) 


This  implies 


P(V  -  v| 8,4)  -  n  p(V  -  v  1 6,4,) 

j  J  J 


(19) 


Using  (19) , 


p(V  =  v|fl)  -  J  p(V  =  vj  d  ,4>)  p(^|fl)  d^ 


-  f  n  p(V  =  V  \6,<t>)  p(*|rf)  d<fi  .  (20) 

j  J  J 


But  for  local  independence  to  hold  in  the  usual  sense  for  the 
multiple -category  model,  it  would  be  necessary  that 


8 


P(V  -  v|«)  -  n  /  P(v  -  v  \e,<t>)  P(*|*)  dd  , 

j  J  J 

and  this  does  not  generally  follow  from  (20)  as  the  order  of 
multiplication  and  integration  has  been  interchanged.  It  does 
follow  if  for  each  9  value,  4>  takes  the  same  value  for  all 
examinees  with  that  6  value;  that  is,  the  variables  of  the 
omitting  process  may  vary  from  one  value  of  9  to  another,  but  not 
among  examinees  with  the  same  value  of  6.  Lord's  objection,  then, 
may  be  stated  as  a  desire  to  allow  for  different  propensities  for 
omitting  to  occur  within  a  given  level  of  ability. 

A  second  reservation  that  might  be  offered  for  this  approach 
stems  from  the  fact  that  probabilities  for  v  given  6  are  averages 
over  4> .  Even  if  (i)  f  (u)  is  an  IRT  model  satisfying  local 

U 

independence  and  (ii)  f  (u)  g  (mlu)  satisfies  extended  local 

V  <P 

* 

independence,  the  multiple-category  response  curves  f.(v)  will 

0 

vary  from  one  group  of  examinees  to  the  next  unless  the 
conditional  distributions  p(^|0)  are  invariant  over  groups. 

How  to  model  omits,  if  vou  must.  Standard  IRT  concerns 
examinees'  tendencies  to  make  correct  responses  when  omits  cannot 
occur.  When  they  can  occur,  the  differences  among  examinees' 
tendencies  to  omit  responses  can  be  cast  as  a  nuisance  variable  in 
the  classic  sense.  It  is  often  easier  to  deal  with  such 
extraneous  influences  at  the  time  the  data  are  collected  than  to 
model  them  after  the  fact.  In  aptitude  and  ability  testing,  we 
should  inform  examinees  as  clearly  as  possible  how  their 
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performance  will  be  evaluated,  and  persuade  them  as  convincingly 
as  possible  to  use  the  omitting  strategy  that  maximizes  their 
expected  score.  To  the  degree  we  succeed,  variation  in  <£  is 
reduced  and  examinees'  data  differ  mainly  because  of  differences 
in  8  .  If  too  the  proportion  of  omits  is  low,  imputing 
fractionally-correct  or  even  random  responses  at  the  level  c 
yields  inferences  that  are  plausible,  readily-calculable ,  and 
robust  with  respect  to  alternative  models  for  omitting. 

For  the  sake  of  completeness,  however,  we  now  outline  an 
approach  using  a  full  model  for  response  and  omission.  The  model 
exhibits  local  independence  for  elements  of  U  given  6,  and 
extended  local  independence  for  elements  of  (U,M)  given  (6  ,<}>)  . 

Its  implementation  requires  either  that  g  is  assumed  to  be  known 
or  that  an  experiment  with  the  same  items  and  similar  examinees 
has  revealed  the  values  of  item  responses  that  were  originally 
omitted.  We  assume  here  that  the  experiment  has  been  carried  out, 
and  a  complete  data  matrix  (u,m)  is  available  for  a  sample  of 
examinees  from  a  population  of  interest. 

An  IRT  model  f  (U  =  u ,6,p)  is  assumed  for  item  responses. 

0 


The  missingness  process  is  modeled  in  terms  of  functions 

g(M.  =  m.lu.  =  u.,d,<b,n.)  for  each  item,  where  n .  are  now 
&  J  J  1  J  J  J  J 

additional  item  parameters  for  the  frequency  with  which  the  item 


is  omitted.  For  example,  we  could  estimate  from  the  completed 


data  set  item-omitting  parameters  -  (d^  ^ ,  e^  ^  ,  d^  ^  ,  e^  that  give 
the  logit  regression  of  nu  on  6  when  u^  -  0  and  when  u^  -  1 ;  that 
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logit  P (Mj  •=  0|u  -  OJ.rip  -  djQ  9  +  e  Q 
logit  P(M  -  0|Uj  -  l.^.iij)  -  d  x  6  +  ejl  , 


where  logit  P  =  log(P/(l  -  P)).  The  examinee  omitting  parameter  <f> 
could  then  be  a  tendency  to  omit  more  or  less  than  average,  so 
that 


logit  P(M 


j 


0|U. 


djo 


+  e 


jo 


+  <t> 


logit  P(M.  -  0 |u 


1,6  ,4>,  rip 


ji 


*  eji  *  * 


The  complete-data  likelihood  function  for  the  item  parameters  is 


L(/9,»?|u,m)  - 


N  -  -  - 

n  J/  fe(ui\e,p)  h(m.|u. p <*,*)  as  d *  .  (21) 


Equation  (21)  provides  a  basis  for  estimating  ( P  ,r) )  from  the 
experimental  data,  either  directly  via  maximum  likelihood  or, 
after  multiplication  by  a  prior  distribution,  via  Bayesian 

A  A 

methods.  Maximum  likelihood  yields  point  estimates  ( p  yr\ ); 
Bayesian  methods  yield  the  posterior  distribution  p(£,r?|u,m). 
Estimating  the  examinee  parameter  distribution  p (8  ,<f>)  at  the  same 
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time  might  also  be  desirable,  say  by  positing  a  functional  form 
and  estimating  its  parameters. 

The  results  of  this  calibration  can  then  be  used  to  estimate 
the  6  value  of  new  examinee  i,  from  whom  only  v^  is  observed. 
Under  Bayesian  inference,  the  relevant  posterior  distribution  is 


p(0|v^,u,m)  « 


JXf  L(vi|5  p(0,»jju,m)  p  (6  ,<t>)  dp  drj  d  <j> 


Under  maximum  likelihood  inference,  the  maximizing  value  (6  ,<j>)  of 

~  A  A 

L(0 ,4>\v  ^,P=P  might  be  sought. 


Inferences  about  Item  Parameters 
When  not  all  item  responses  are  observed,  the  (marginal) 
likelihood  function  for  item  parameters  P  induced  by  the  data 
matrix  v  =  (v^, . . . ,v^)  from  a  sample  of  N  examinees  is 


L(0|v) 


N 


=  n  J7J  p(u 


mis 


i 


,u 


obs ,  i 


e,<t>,P)  dumis  .  p (e,4>)  ae  a<f> 


N 

=  n  JJ  pcvjtf,*,^)  P ae  a*  , 


(22) 


52 


where  u  ,  ,  and  u  .  .  are  the  observed  and  missing  portions  of 

obs,i  mis.i  6  r 

the  response  vector  of  examinee  i.  The  psuedo-likelihood  obtained 
by  ignoring  the  missingness  process  is 

*  N 

L  (^"obs>  -  "  I  P<uobs  ll*  ,0)  p(tf)  d8  .  (23) 

i 

Ignoring  the  missingness  process  when  making  direct 
likelihood  inferences  about  0  means  comparing  the  values  of  (23) 
rather  than  (22)  at  different  values  of  0.  Equation  (22)  differs 
from  (23)  by  integrating  over  P(UQ^S  with  respect  to  6  for 

each  examinee,  rather  than  over  p(v^|fl,^,/))  with  respect  to  6  and 
<t> .  The  resulting  integrals  are  proportional  with  respect  to  0  if 
and  only  if  for  all  values  of  0,  the  conditions  for  ignorability 
for  9  given  0  are  satisfied  for  Bayesian  inference.  Therefore, 

o  Ignorability  under  direct  (marginal)  likelihood  inference 
about  0  holds  if  (i)  ignorability  under  Bayesian  inference 
holds  for  each  B  conditional  on  0,  and  (ii)  0,  9,  and  <)>  are 
distinct  in  the  sense  required  for  direct  likelihood 
inference . 

If  0  is  a  priori  independent  of  all  6  and  4> ,  the  correct 
Bayesian  posterior  for  0  in  the  presence  of  missing  data  is  the 
product  of  (22)  and  the  prior  p(^) .  Ignoring  the  missingness 
process  when  making  Bayesian  inferences  about  0  means  using 
instead  the  product  of  (23)  and  p(0) .  The  preceding  result 
indicates  when  ignoring  the  missingness  gives  the  correct 
likelihood.  To  obtain  the  correct  posterior,  then,  we  have: 
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o 


Ignorability  holds  under  Bayesian  inference  about  p  if  (i) 
ignorability  holds  under  Bayesian  inference  for  each  6 
conditional  on  P,  (ii)  p,  6,  and  <f>  are  distinct  in  the  sense 
required  for  Bayesian  inference. 

'fc 

If  ignorability  holds  under  direct  likelihood  inference,  L 

A 

yields  the  correct  value  for  p.  The  necessary  condition  for 

A 

sampling  distribution  inferences  based  on  p  requires  that  the 
probability  of  the  missin^ness  pattern- -in  this  context,  the 
distribution  of  counts  of  individual  missingness  patterns- -not 
depend  on  the  values  of  observed  responses.  This  condition  is 
implied  by  MCAR.  When  it  and  direct-likelihood  ignorability  hold, 
conditional  sampling  distribution  inferences  are  appropriate. 

They  pertain  to  sampling  of  item  responses  to  the  observed  items 
from  repeated  subsamples  of  examinees  with  each  observed 
missingness  pattern,  with  subsample  sizes  fixed  at  the  observed 
counts  of  those  missingness  patterns. 

Case  1 .  We  have  seen  that  for  alternate  test  forms, 
ignorability  holds  for  Bayesian  inferences  about  6  given  p. 

Random  assignment  of  test  forms  ensures  MCAR.  Therefore  the 
responses  to  items  on  forms  that  are  not  administered  can  be 
ignored  under  direct  likelihood  inference  about  P,  and  under 
Bayesian  inferences  as  well  as  long  as  prior  distributions  are 
independent.  Sampling-distribution  inferences  can  based  on  the 

A 

MLE  P  with  the  understanding  that  they  pertain  to  repeated 
sampling  of  examinees  for  each  form  in  the  sample  sizes  that  were 
actually  observed. 
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Case  2 .  In  targeted  testing,  items  believed  to  be  easier 
are  administered  to  examinees  that  are  expected  to  have  lower 
abilities,  and  items  believed  to  be  harder  are  administered  to 
examinees  expected  to  have  higher  abilities.  Bayesian  inferences 
about  9  given  ft  can  ignore  missingness  only  after  conditioning  on 
y,  the  collateral  examinee  variable  used  in  test-form  assignment. 
Correct  inferences  about  ft  under  direct  likelihood  inference  thus 
require  that  p(0)  in  (23)  be  replaced  by  p(#|y^),  as  well  as  that 
all  values  of  ft  are  possible  (if  not  always  likely)  on  all  test 
forms.  Bayesian  inferences  must  additionally  take  into  account 
the  prior  beliefs  that  led  to  the  differential  assignment  of  items 
to  forms.  Let  z  «*  (z^,...,z  )  represent  the  collateral 
information  about  items  used  to  make  these  assignments  (e.g., 
pretest  item  difficulties  or  item  content).  Appropriate  Bayesian 
inferences  about  ft  that  account  for  the  missingness  process  may  be 
drawn  from 


p(^Uobs’y,z)  “  L*(^lUobs’y)  p(^lz)l 

As  in  Case  1,  the  distribution  of  counts  of  missingness  patterns 
does  not  depend  on  the  values  of  observed  item  responses,  so 
conditional  sampling-distribution  inferences  about  ft  from 
L  (/j|uQks  ,y)  are  appropriate.  They  pertain  to  repeated 
administrations  of  the  observed  counts  of  administered  forms  at 
each  value  of  y,  to  samples  of  examinees  with  those  y  values. 
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Case  3 .  Conclusions  similar  to  those  of  Case  2  hold  for 


direct  likelihood  and  Bayesian  inference  in  adaptive  testing. 
Ignorability  holds  for  Bayesian  inference  about  6  given  /9  (again 
conditioning  on  y  if  collateral  examinee  variables  are  used  in 
item  selection) ,  so  direct  likelihood  inferences  about  &  from  L 
are  justified  (Verhelst  and  Veldhuijzen,  1987) - -though  not  always 
satisfactory.  The  reason  is  that  p(^|z)  is  often  very  strong  in 
practice;  indeed,  f)  is  usually  treated  as  known  for  the  purpose  of 


item  selection.  In  this  case,  z.  could  be  the  mean  B.  and 

J  J 


covariance  matrix  of  an  assumed  normal  distribution  for  B . ,  and 

J 


item  selection  would  be  based  on  B . .  The  data  collected  for  a 

J 


given  item  when  it  is  administered  adaptively  tend  to  be  from 
examinees  in  a  relatively  narrow  band  of  ability.  For  binary 
items  with  more  than  one  parameter,  the  number  of  examinees 
required  for  stable  estimates  may  well  exceed  the  number  that  can 
be  tested  in  practice.  Bayesian  inference  is  preferable  under 
these  circumstances.  Provisional  item  parameter  estimates  based 
on  p(/}|z)  may  be  used  to  administer  items,  then 
adaptively-acquired  item  responses  can  be  used  to  produce  an 
updated  distribution 


p(^l2,Uobs)  a  L  (u-hJ^)  P(^lz)  ■ 


obs  1 


Because  the  missingness  process  is  ignorable  under  direct 


likelihood  inference,  the  usual  MLE  obtained  by  maximizing  L 


gives  the  correct  point  estimate  for  MLE-based  sampling- 
distribution  inferences  about  fi.  As  in  the  section  on  estimating 
8  from  adaptive  tests,  however,  the  necessary  condition  for 
ignorability  under  sampling-distribution  inference  is  not 
satisfied- -  the  probabilities  of  missingness  patterns  depends  on 

*ic 

the  values  of  observed  responses.  MLE  properties  based  on  L  need 
not  apply  to  item  parameter  estimates  obtained  from  adaptive  test 
data.  It  may  be  that  for  some  adaptive  test  designs,  the  usual 
variance  estimates  with  m  fixed  at  m  are  good  approximations  to 
the  variances  that  would  be  obtained  under  repeated  sampling  of 
the  entire  adaptive  test  for  N  examinees,  but  this  must  be 
determined  case  by  case. 

Case  4.  Recall  that  when  some  items  at  the  end  of  a  test  are 
not  reached,  MAR  holds  for  inferences  about  8  given  fi  but  Bayesian 
ignorability  does  not  hold  unless  speed  and  ability  are 
independent.  Missingness  due  to  time  limitations,  therefore,  is 
not  generally  ignorable  under  any  type  of  inference  about  fi. 
Assuming  there  are  no  restrictions  on  the  parameter  space,  drawing 
likelihood  inferences  about  fi  requires  one  to  replace  p(0)  in  (22) 
with  p(0|nu),  where 

p(0  |nu )  a  p (8)  f  p(M  -  mj^)  p(^|fl)  d<f>  . 

If,  in  addition,  the  test  has  been  assembled  to  start  with  easy 
items  and  become  harder,  the  prior  information  about  items  [say 


Case  5 .  The  topic  of  inference  about  item  parameters  when 
examinees  omit  some  items  intentionally  has  already  been  broached 
in  the  discussion  about  estimating  6.  Bayesian  ignorability  for 
8  given  0  does  not  generally  hold,  so  missingness  mechanisms  must 
be  specified  and  inferences  about  0  must  start  with  on  the  full 
likelihood  (23).  A  number  of  approaches  were  discussed  there, 
including  imputing  responses  for  omits,  using  a  multiple-category 
IRT  model,  and  fitting  Lord's  (1983)  model  for  responses  and 
omits.  The  approach  that  is  most  easily  incorporated  into 
standard  IRT  algorithms  is  to  treat  intentional  omits  as 
fractionally  correct  (Lord,  1974) .  Assuming  that  examinees  omit 
only  in  accordance  with  the  strategy  that  maximizes  their  expected 
score,  this  approach  gives  "marginal-conditional"  MLEs  that 
maximize 
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n  f  n 

i  obs . 

1 


u. .  1-u • 

Pj(0)  1JQ..(0) 


n  P.(*)CQ .(tf)1'6 

mis.  ^  J 

1 


p(S)  &6 
(24) 


Equation  (24)  is  "conditional"  in  that  it  accounts  for  the 
influence  of  8  upon  item  responses  given  the  pattern  of 
missingness,  but  does  not  capture  the  role  of  8  in  determining 
that  pattern.  It  is  "marginal"  in  that  it  is  the  expectation  over 
U  .  given  u  ,  and  m  of  the  conditional  likelihood  L  (fllu)  that 
would  be  maximized  if  all  responses  had  been  observed. 

Summary 

In  practical  applications  of  item  response  theory  (IRT) , 
there  are  several  reasons  that  item  responses  may  not  be  observed 
from  all  examinees  to  all  test  items.  Ignoring  the  missingness 
process  under  direct  likelihood  inference  means  using  a  psuedo- 
likelihood  that  includes  terms  for  only  the  responses  that  were 
observed,  without  regard  for  the  processes  by  which  they  came  to 
be  observed.  The  resulting  inferences  are  appropriate  if  the 
psuedo- likelihood  is  proportional  to  the  correct  likelihood  that 
does  account  for  the  missingness  process.  In  this  case  the 
correct  point  estimate  of  an  MLE  is  obtained.  Sampling- 
distribution  inferences  from  the  MLE  are  appropriate  only  if  the 
missingness  pattern  does  not  depend  on  the  values  of  the  observed 
data.  When  this  condition  holds,  sampling-distribution  inferences 
can  be  drawn  with  regard  to  repeated  samples  of  responses  under 
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the  observed  pattern  of  missingness.  The  missingness  process  is 
ignorable  with  respect  to  Bayesian  inference  if  the  correct 
Bayesian  posterior  is  proportional  to  the  product  of  the 
psuedo-likelihood  and  an  appropriate  prior  distribution. 

For  fives  common  types  of  missingness  in  IRT,  we  used  Rubin's 
(1976)  theorems  to  determine  whether  ignorability  holds  under 
direct  likelihood  and  Bayesian  inference  about  examinee  parameters 
6  when  item  parameters  fi  are  known.  In  those  cases  in  which  the 
correct  value  of  the  MLE  is  obtained  under  direct  likelihood 
inference,  we  asked  whether  sampling  distribution  inferences  based 
on  the  MLE  were  appropriate.  We  then  considered  the  analogous 
questions  for  inferences  about  f)  when  the  examinee  parameters  are 
eliminated  by  marginalization.  Our  findings  are  summarized  below. 
Tables  1  and  2  highlight  the  results  on  ignorability. 

Case  1:  Alternate  test  forms.  When  an  examinee  is  assigned  one  of 
several  alternative  test  forms  by  a  random  process  such  as  a  coin 
flip  or  a  spiralling  scheme,  the  process  that  renders  missing  the 
responses  to  items  on  the  forms  not  presented  is  ignorable  for  all 
three  types  of  inference,  both  for  estimating  p  and  for  estimating 
8  when  P  is  known. 

Case  2:  Targeted  testing.  When  collateral  variables  such  as 
educational  or  demographic  status  are  used  to  assign  an  examinee 
one  of  several  tests  that  differ  in  their  measurement  properties, 
the  resulting  missingness  on  forms  not  given  is  ignorable  under 
direct  likelihood  inference  for  8  given  0,  but  not  under  Bayesian 
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n 


inference  unless  the  prior  information  about  examinees  that  led  to 
differential  assignments  is  conditioned  on.  This  information  must 
be  taken  into  account  for  both  likelihood  and  Bayesian  inferences 
about  0\  for  Bayesian  inference,  prior  information  about  0  used  to 
select  items  must  additionally  be  taken  into  account.  Sampling 
distribution  inferences  may  be  based  on  MLEs  for  0  and  for  0  given 
0,  conditional  on  the  observed  patterns  of  test  form 
administration  within  values  of  the  examinee  variables  used  for 
targeting. 

Case  3:  Adaptive  testing.  The  same  conclusions  for  direct 
likelihood  and  Bayesian  inference  follow  in  adaptive  testing, 
where  assignment  proceeds  item  by  item  in  accordance  with  the 
values  of  responses  to  preceding  items.  Ignorability  under  direct 
likelihood  inference  means  that  the  correct  points  are  identified 
as  MLEs  of  0  given  0  and  of  0,  but  the  usual  MLE  properties  under 
sampling-distribution  inference  need  not  hold  because  the 
probabilities  of  missingness  patterns  depend  on  the  values  of 
observed  responses. 

Case  4:  Not- reached  items.  When  some  examinees  do  not  interact 
with  the  last  items  on  a  nearly  nonspeeded  test,  the  not-reached 
process  is  ignorable  with  respect  to  direct  likelihood  inference 
about  0  given  0,  and  the  MLE  supports  sampling  distribution 
inferences  that  pertain  to  repeated  administrations  of  the  items 
that  were  actually  reached.  This  missingness  process  is  not 
ignorable  under  Bayesian  inference  unless  speed  and  ability  are 
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independent.  And  only  then  can  direct  (marginal)  likelihood 
inferences  about  /3  ignore  the  missingness.  Bayesian  inferences 
about  further  require  that  prior  knowledge  about  item  parameters 
be  employed  if  it  played  a  role  in  determining  which  items  would 
not  be  reached,  as  when  items  are  ordered  from  easy  to  hard. 

Case  5:  Omitted  items.  When  examinees  are  presented  items,  have  a 
chance  to  appraise  their  content,  and  decide  for  their  own  reasons 
not  to  respond,  the  missingness  is  not  ignorable.  Inferences  must 
be  drawn  from  a  full  model  for  the  joint  distribution  of 
missingness  and  item  response. 

Not  surprisingly,  modeling  this  nonignorable  nonresponse  is 
difficult.  Neither  of  the  two  most  ambitious  approaches  proposed 
to  date,  namely  Lord's  (1983)  model  for  omits  and  the  use  of 
multiple-category  IRT  models,  handles  the  issue  of  local 
independence  in  a  fully  satisfactory  manner.  Under  the 
assumption  that  examinees  are  perfect  judges  of  their  chances  of 
responding  correctly,  and  omit  only  if  it  is  in  accordance  with 
the  strategy  that  maximizes  their  expected  score,  Lord's  (1974) 
treatment  of  omits  as  fractionally  correct  can  be  justified  as 
providing  the  expectation  of  a  conditional  term  in  the  full 
likelihood.  This  procedure  is  readily  incorporated  into  standard 
complete-data  IRT  algorithms  and  avoids  having  to  specify  the 
full  likelihood,  but  sacrifices  information  about  examinee  and 
item  parameters  conveyed  by  the  observed  pattern  of  missingness. 
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Table  1 


Ignorability  Results  for  Estimating  6  Given  /3 


Type  of  Inference 

Type  of  . - . . 

Missingness  Direct  Likelihood  Bayesian  Sampling  Distribution 


Alternate 

Yes 

Yes 

Yes 

Forms 

Targeted 

Forms 

Yes 

Yes,  given 
examinee  variables 

Yes 

Adaptive 

Testing 

Yes 

Yes ,  given 
examinee  variables 
if  they  are  used 

No 

Not-Reached 

Yes 

No,  unless  speed  and 
ability  are  independent 

Yes 

Intentional 

Omissions 

No 

No 

No 

Conditional  on  the  observed  pattern  of  missingness. 
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Table  2 


Ignorability  Results  for  Estimating  /9  After  Marginalizing  over  9 


Type  of 
Missingness 

ii 

1 

i 

i 

J 

i 

Type  of  Inference 

n 

n 

a 

i 

! 

a 

a 

Direct  Likelihood 

■A- 

Bayesian  Sampling  Distribution 

Alternate 

Forms 

Yes 

Yes 

Yes 

Targeted 

Forms 

Yes,  given 
examinee  variables 

Yes,  given 
examinee  and  item 
variables 

Yes ,  given 
examinee  variables 

Adaptive 

Testing 

Yes ,  given 
examinee  variables 
if  they  are  used 

Yes,  given 
item  variables  and 
examinee  variables 
if  they  are  used 

No 

Not-Reached 

No,  unless  speed 
and  ability  are 
independent 

No,  unless  speed 
and  ability  are 
independent 

No,  unless  speed 
and  ability  are 
independent 

Intentional 

Omissions 

No 

No 

No 

Conditional  on  the  observed  pattern  of  missingness. 
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AFHRL/MOEL 

Brooks  AFB,  TX  78235 

Dr.  Norman  Cliff 
Department  of  Psychology 
Un i v .  of  So.  California 
Los  Angeles,  CA  90089-1061 
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D i rector, 

Manpower  Support  and 
Readiness  Program 
Center  for  Naval  Analysis 
2000  North  Beauregard  Street 
Alexandria,  VA  22311 

Dr.  Stan  ley  Co  I  I yer 
Office  of  Naval  Technology 
Code  222 

800  N.  Quincy  Street 
Arl ington,  VA  22217-5000 

Dr.  Hans  F.  Crombag 
Facu I ty  of  Law 
University  of  Limburg 
P.0.  Box  616 
Maastr i cht 

The  NETHERLANDS  6200  MD 

Dr.  Timothy  Davey 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  C.  M.  Dayton 
Department  of  Measurement 
Statistics  4  Evaluation 
Co  I  I ege  of  Educat i on 
University  of  Maryland 
Col  lege  Park,  MD  20742 

Dr.  Ralph  J.  DeAyala 
Measurement,  Statistics, 
and  Evaluation 
Benjamin  Bldg.,  Rm.  4112 
University  of  Maryland 
College  Park,  MD  20742 

Dr.  Dattprasad  Divgi 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  Hei-Ki  Dong 
Bell  Communications  Research 
6  Corporate  Place 
PVA-1K226 

Piscataway,  NJ  08854 


Dr .  Fritz  Drasgow 
University  of  Illinois 
Department  of  Psychology 
603  E.  Daniel  St. 

Champaign,  IL  61820 

Defense  Technical 

Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22314 
Attn:  TC 
(12  Cop  i  es> 

Dr.  Stephen  Dunbar 
224B  Lindquist  Center 
for  Measurement 
University  of  I owa 
Iowa  City,  I A  52242 

Dr.  James  A.  Earles 

Air  Force  Human  Resources  lab 

Brooks  AFB,  TX  78235 

Dr.  Kent  Eaton 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  John  M.  Eddins 
University  of  Illinois 
252  Engineering  Research 
Laboratory 

103  South  Mathews  Street 
Urbana,  IL  61801 

Dr.  Susan  Embretson 
University  of  Kansas 
Psychology  Department 
426  Fraser 
Lawrence,  KS  66045 

Dr.  George  Englehard,  Jr. 
Division  of  Educational  Studies 
Emory  University 
210  Fishburne  Bldg. 

Atlanta,  GA  30322 

Dr.  Benjamin  A.  Fairbank 
Performance  Metrics,  Inc. 

5825  Ca I  I aghan 
Suite  225 

San  Antonio,  TX  78228 


1988/07/06 
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Dr.  P-A.  Federico 

Code  51 

NPRDC 

San  Diego,  CA  92152-6800 

Dr.  Leonard  Feldt 
L i ndqu i st  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  I A  52242 

Dr.  Richard  L.  Ferguson 
American  College  Testing 
P.0.  Box  168 
Iowa  City,  IA  52243 

Dr.  Gerhard  Fischer 
L i eb i ggasse  5/3 
A  1010  Vienna 
AUSTRIA 

Dr.  Myron  Fischl 
U.S.  Army  Headquarters 
DAPE-MRR 
The  Pentagon 

Washington,  DC  20310-0300 

Prof.  Donald  Fitzgerald 
University  of  New  England 
Department  of  Psychology 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

Mr.  Paul  Foley 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Or .  A I f  red  R.  Freg I y 
AFOSR/NL ,  Bldg.  410 
Bolling  AFB,  DC  20332-6448 

Dr.  Robert  D.  Gibbons 
Illinois  State  Psychiatric  Inst. 
Rm  529W 

1601  W.  Taylor  Street 
Chicago,  IL  60612 

Dr.  Janice  Gifford 
University  of  Massachusetts 
School  of  Education 
Amherst,  MA  01003 


Dr.  Robert  Gl aser 
Learning  Research 

&  Development  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

Dr.  Bert  Green 
Johns  Hopkins  University 
Department  of  Psychology 
Charles  &  34th  Street 
Baltimore,  MD  21218 

DORNIER  GMBH 
P.0.  Box  1420 
D-7990  Fr i edr i chshaf en  1 
WEST  GERMANY 

Dr.  Ronald  K.  Hambleton 
University  of  Massachusetts 
Laboratory  of  Psychometric 
and  Evaluative  Research 
Hills  South,  Room  152 
Amherst,  MA  01003 

Dr.  Delwyn  Harnisch 
University  of  Illinois 
51  Gerty  Drive 
Champaign,  IL  61820 

Dr.  Grant  Henning 
Senior  Research  Scientist 
Division  of  Measurement 
Research  and  Services 
Educational  Testing  Service 
Princeton,  NJ  08541 

Ms.  Rebecca  Hetter 
Navy  Personnel  R&D  Center 
Code  63 

San  Diego,  CA  92152-6800 

Dr.  Pau I  W.  Ho  I  I  and 
Educational  Testing  Service,  2 1  — T 
Rosedale  Road 
Princeton,  NJ  08541 

Prof.  Lutz  F.  Hornke 
Institut  fur  Psychologie 
RWTH  Aachen 
Jaegerstrasse  17/19 
D-5100  Aachen 
WEST  GERMANY 


1988/07/06 
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Dr.  Paul  Horst 
677  G  Street,  #184 
Chula  Vista,  CA  92010 

Mr.  Dick  Hoshaw 
OP- 135 

Ar I i ngton  Annex 
Room  2834 

Washington,  DC  20350 

Dr.  Lloyd  Humphreys 
University  of  Illinois 
Department  of  Psychology 
603  East  Daniel  Street 
Champaign,  IL  61820 

Dr.  Steven  Hunka 
3-104  Educ.  N. 

University  of  Alberta 
Edmonton,  Alberta 
CANADA  T6G  2G5 

Or.  Huynh  Huynh 
College  of  Education 
Univ.  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Robert  Jannarone 
Elec,  and  Computer  Eng.  Dept. 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Douglas  H.  Jones 
Thatcher  Jones  Associates 
P.0.  Box  6640 
10  Trafalgar  Court 
Lawrencev i I  I e ,  NJ  08648 

Dr.  Mil  ton  S.  Katz 
European  Science  Coordination 
Off i ce 

U.S.  Army  Research  Institute 
Box  65 

FPO  New  York  09510-1500 

Prof.  John  A.  Keats 
Department  of  Psychology 
University  of  Newcastle 
N.S.W.  2308 
AUSTRALIA 


Dr.  G.  Gage  Kingsbury 

Portland  Public  Schools 

Research  and  Evaluation  Department 

501  North  Dixon  Street 

P.  0.  Box  3107 

Portland,  OR  97209-3107 

Dr.  William  Koch 
Box  7246,  Meas.  and  Eval.  Ctr. 
University  of  Texas-Austin 
Austin,  TX  78703 

Dr.  James  Kraatz 
Computer-based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Leonard  Kroeker 
Navy  Personnel  R&D  Center 
Code  62 

San  Diego,  CA  92152-6800 

Dr.  Jerry  Lehnus 

Defense  Manpower  Data  Center 

Su i te  400 

1600  Wi Ison  Bl vd 

Rosslyn,  VA  22209 

Dr.  Thomas  Leonard 
University  of  Wisconsin 
Department  of  Statistics 
1210  West  Dayton  Street 
Madison,  WI  53705 

Dr.  Michael  Levine 
Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

Dr.  Charles  Lewis 
Educational  Testing  Service 
Princeton,  NJ  08541-0001 

Dr.  Robert  L.  Linn 
Campus  Box  249 
University  of  Colorado 
Boulder,  CO  80309-0249 


1988/07/06 
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Or.  Robert  Lockman 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  Frederic  H.  Lord 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  George  B.  Macready 
Department  of  Measurement 
Statistics  &  Evaluation 
College  of  Education 
University  of  Maryland 
College  Park,  MD  20742 

Dr.  Gary  Marco 
Stop  31-E 

Educational  Testing  Service 
Princeton,  NJ  08451 

Dr.  James  R.  McBride 
The  Psychological  Corporation 
1250  Sixth  Avenue 
San  Diego,  CA  92101 

Dr.  Clarence  C.  McCormick 
HQ,  USMEPCOM/MEPCT 
2500  Green  Bay  Road 
North  Chicago,  IL  60064 

Dr.  Robert  McKinley 
Educational  Testing  Service 
16-T 

Princeton,  NJ  08541 

Dr.  James  McMichael 
Technical  Director 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Barbara  Means 
SRI  International 
333  Ravenswood  Avenue 
Menlo  Park,  CA  94025 

Dr.  Robert  M i s I evy 
Educational  Testing  Service 
Princeton,  NJ  08541 


Dr.  William  Montague 

NPRDC  Code  13 

San  Diego,  CA  92152-6800 

Ms.  Kathleen  Moreno 
Navy  Personnel  R&D  Center 
Code  62 

San  Diego,  CA  92152-6800 

Headquarters  Marine  Corps 
Code  MPI-20 
Washington,  DC  20380 

Dr.  W.  Alan  Nicewander 
University  of  Oklahoma 
Department  of  Psychology 
Norman,  OK  73071 

Deputy  Technical  Director 

NPRDC  Code  01A 

San  Diego,  CA  92152-6800 

Director,  Training  Laboratory, 
NPRDC  (Code  05) 

San  Diego,  CA  92152-6800 

Director,  Manpower  and  Personnel 
Laboratory , 

NPRDC  (Code  06) 

San  Diego,  CA  92152-6800 

Director,  Human  Factors 

&  Organizational  Systems  Lab, 
NPRDC  (Code  07) 

San  Diego,  CA  92152-6800 

Library,  NPRDC 
Code  P201L 

San  Diego,  CA  92152-6800 

Commanding  Officer, 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

Dr.  Harold  F.  O'Neil,  Jr. 

School  of  Education  -  WPH  801 
Department  of  Educational 
Psychology  &  Technology 
University  of  Southern  California 
Los  Angeles,  CA  90089-0031 


1988/07/06 


Educational  Testing  Serv i ce/Mi s I evy 


Or.  James  B.  Olsen 
WICAT  Systems 
1875  South  State  Street 
Orem,  UT  84058 

Office  of  Naval  Research, 

Code  1142CS 
800  N.  Quincy  Street 
Arl ington,  VA  22217-5000 
<6  Copies) 

Office  of  Naval  Research, 

Code  125 

800  N.  Quincy  Street 
Arlington,  VA  22217-5000 

Assistant  for  MPT  Research, 
Development  and  Studies 
OP  01B7 

Washington,  DC  20370 

Dr.  Jud i th  Orasanu 
Basic  Research  Office 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  J^sse  Orlansky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 
Alexandria,  VA  22311 

Dr.  Randolph  Park 
Army  Research  Institute 
5001  Eisenhower  Blvd. 

Alexandria,  VA  22333 

Wayne  M.  Patience 
American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Circle,  NW 
Washington,  DC  20036 

Dr.  James  Paulson 
Department  of  Psychology 
Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

Dept,  of  Administrative  Sciences 
Code  54 

Naval  Postgraduate  School 
Monterey,  CA  93943-5026 


Department  of  Operations  Research, 
Naval  Postgraduate  School 
Monterey,  CA  93940 

Dr.  Mark  D.  Reckase 
ACT 

P.  0.  Box  168 
Iowa  City,  IA  52243 

Dr .  Malcolm  Ree 
AFHRL/MOA 

Brooks  AFB,  TX  78235 

Dr.  Barry  Riegelhaupt 
HumRRO 

1100  South  Washington  Street 
Alexandria,  VA  22314 

Dr.  Carl  Ross 
CNET-PDCD 
Building  90 

Great  Lakes  NTC,  IL  60088 
Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Fumiko  Samejima 
Department  of  Psychology 
University  of  Tennessee 
310B  Austin  Peay  Bldg. 

Knoxville,  TN  37916-0900 

Mr.  Drew  Sands 

NPRDC  Code  62 

San  Diego,  CA  92152-6800 

Lowe  I  I  Schoer 

Psychological  &  Quantitative 
Foundat i ons 
College  of  Education 
University  of  Iowa 
Iowa  City,  IA  52242 

Dr.  Mary  Schratz 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Dr .  Dan  Sega  I  I 

Navy  Personnel  R8tD  Center 

San  Diego,  CA  92152 


1988/07/06 


Educational  Testing  Serv i ce/M i s I evy 


Dr.  W.  Steve  Se I Iman 
CA3D (MPA&L  > 

28269  The  Pentagon 
Washington,  DC  20301 

Dr.  Kazuo  Shigemasu 
7-9-24  Kugenuma-Ka i gan 
Fuj i sawa  251 
JAPAN 

Dr .  William  Sims 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  H.  Wallace  Sinaiko 
Manpower  Research 

and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street,  Suite  120 
Alexandria,  VA  22314-1713 


Or.  Hariharan  Swaminathan 
Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

Mr.  Brad  Sympson 

Navy  Personnel  R&D  Center 

Code-62 

San  Diego,  CA  92152-6800 

Dr.  John  Tangney 
AFOSR/NL ,  Bldg.  410 
Bolling  AFB,  DC  20332-6448 

Dr.  Kikumi  Tatsuoka 
CERL 

252  Engineering  Research 
Laboratory 

103  S.  Mathews  Avenue 
Urbana,  IL  61801 


Dr.  Richard  E.  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 


Dr.  Maurice  Tatsuoka 
220  Education  Bldg 
1310  S.  Sixth  St. 
Champaign,  IL  61820 


Dr.  Richard  C.  Sorensen 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Paul  Speckman 
University  of  Missouri 
Department  of  Statistics 
Columbia,  MO  65201 

Dr.  Judy  Spray 
ACT 

P.0.  Box  168 
Iowa  City,  IA  52243 

Dr.  Martha  Stocking 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr .  William  Stout 
University  of  Illinois 
Department  of  Statistics 
101  1 1  I  ini  Hall 
725  South  Wright  St. 
Champaign,  IL  61820 


Dr.  David  Thissen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

Mr.  Gary  Thomasson 
University  of  Illinois 
Educational  Psychology 
Champaign,  IL  61820 

Dr.  Robert  Tsutakawa 
University  of  Missouri 
Department  of  Statistics 
222  Math.  Sciences  Bldg. 
Columbia,  MO  65211 


Dr.  Ledyard  Tucker 
University  of  Illinois 
Department  of  Psychology 
603  E.  Daniel  Street 
Champaign,  IL  61820 


Educational  Testing  Serv i ce/M i s I evy 


Dr.  Vern  W.  Urry 
Personnel  R&D  Center 
Office  of  Personnel  Management 
1900  E.  Street,  NW 
Washington,  DC  20415 

Dr .  David  Vale 
Assessment  Systems  Corp. 

2233  University  Avenue 

Su i te  440 

St.  Paul ,  MN  55114 

Dr .  F rank  L  .  V i c i no 
Navy  Personnel  R&O  Center 
San  Diego,  CA  92152-6800 

Dr.  Howard  Wainer 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  Ming-Mei  Wang 
L i ndqu i st  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  IA  52242 

Dr.  Thomas  A.  Warm 
Coast  Guard  Institute 
P.  0.  Substation  18 
Oklahoma  City,  OK  73169 

Dr.  Brian  Waters 
HumRRO 

12908  Argy I e  Circle 
Alexandria,  VA  22314 

Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455-0344 

Dr.  Ronald  A.  We i tzman 
Box  146 

Carmel,  CA  93921 

Major  John  Welsh 

AFHRL/MOAN 

Brooks  AFB,  TX  78223 


Dr.  Douglas  Wetzel 
Code  51 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Rand  R.  Wi I  cox 
University  of  Southern 
Cal i f orn i a 

Department  of  Psychology 
Los  Angeles,  CA  90089-1061 

German  Military  Representative 
ATTN:  Wolfgang  Wildqrube 
Stre i tk  raef teamt 
D-5300  Bonn  2 

4000  Brandywine  Street,  NW 
Washington,  DC  20016 

Dr .  Bruce  Willi ams 
Department  of  Educational 
Psycho  I ogy 

University  of  Illinois 
Urbana,  IL  61801 

Dr.  Hilda  Wing 
NRC  MH-176 

2101  Constitution  Ave. 
Washington,  DC  20418 

Dr.  Martin  F.  Wiskoff 
Defense  Manpower  Data  Center 
550  Camino  El  Estero 
Suite  200 

Monterey,  CA  93943-3231 

Mr.  john  H.  Wolfe 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Dr.  George  Wong 
Biostatistics  Laboratory 
Memorial  S I oan-Ketter i ng 
Cancer  Center 
1275  York  Avenue 
New  York,  NY  10021 

Dr.  Wallace  Wulfeck,  III 
Navy  Personnel  R&D  Center 
Code  51 

San  Diego,  CA  92152-6800 
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Or.  Kentaro  Yamamoto 
03-T 

Educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  08541 

Dr.  Wendy  Yen 
CTB/McGraw  Hill 
Oel  Monte  Research  Park 
Monterey,  CA  93940 

Dr.  Joseph  L.  Young 
National  Science  Foundation 
Room  320 

1800  G  Street,  N.W. 
Washington,  DC  20550 

Mr.  Anthony  R.  Zara 
National  Council  of  State 
Boards  of  Nursing,  Inc. 
625  North  Michigan  Avenue 
Suite  1544 
Chicago,  IL  60611 

Dr.  Peter  Stoloff 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 


